o I. il@sBdZddlZddlZddlZddlmZmZmZmZddl m Z m Z m Z m Z mZddlmZmZddlZddlZddlmZddlmZddlZddlZddlZejejdee Z!e d d gd Z"Gd d d eZ#GdddeZ$GdddeZ%GdddeZ&GdddeZ'GdddeZ(GdddeZ)GdddeZ*GdddeZ+GdddeZ,d e-d!e-d"efd#d$Z.d%e-d"e/fd&d'Z0d(e-d"e/fd)d*Z1d+e-d"e2fd,d-Z3ed.Z4e4j5d/d/d0d"e-fd1d2Z6d3e-d e-d4e-d5e-fd6d7Z7d3e-d"eefd8d9Z8dwd3e-d:e-d;efdd?Z:d@e-d e-dAe-d"ee-effdBdCZ;  Ddxd@e-d e-dAe-dEeee-dFe-d"ee-eff dGdHZe"j?dNe*dOe dPdQfdRe#dSedTe-fdUdVZ@e"j?dWe*dOe dXdQfdRe%dSedTe-fdYdZZAe"Bd[d3e-fd\d]ZCe"j?d^e+dOe d_dQfdRe$dTe-fd`daZDe"j?dbe,dOe dcdQfdRe&dTe-fdddeZEe"Bdfe dgdQfd e-dTe-fdhdiZFe"Bdje dkdQfd e-d!e-dle-dTe-fdmdnZGe"?doe dpdQfdRe(dTe-fdqdrZHe"?dse dtdQfdRe)dTe-fdudvZIdS)yza Bid Processing API Routes for Minaions Tender System To be imported and included in main app.py N)DictListOptionalAny) APIRouter HTTPExceptionDependsHeaderBackgroundTasks) BaseModelField)datetime)Path)levelz/api/bid-processingzBid Processing)prefixtagsc@JeZdZUedddZeed<edddZeed<eddd Zeed <d S) ProcessGeMBidsRequest.$Path to input gem_bids_complete.json descriptioninput_file_path Tenant ID tenant_idprocessed_bids_data.jsonOutput filenamedefaultroutput_file_nameN) __name__ __module__ __qualname__r rstr__annotations__rrr$r$>/var/www/html/minaions-tender/ai-engine/bid_processing_apis.pyr" rc@szeZdZUedddZeed<edddZeed<eddd Zeed <ed d dZ e e eed <eddd Z eed<d S)AnalyzeSellersRequest.z Path to processed bids JSON filerrrrseller_analysis_output.jsonrrrNzKeywords to filter bidssearch_keywordsanySearch mode: 'any' or 'all' search_mode) rr r!r rr"r#rrr)rrr,r$r$r$r%r''s r'c@r) ProcessEProcBidsRequest.z%Root directory containing bid foldersrroot_directoryrrrrrrN) rr r!r r.r"r#rrr$r$r$r%r-.r&r-c@s6eZdZUedddZeed<edddZeed<dS)GetBidAnalyticsRequest.z!Path to seller analysis JSON fileranalysis_file_pathrrN)rr r!r r0r"r#rr$r$r$r%r/3s r/c@sbeZdZUedddZeed<edddZeed<eddd Zeed <ed d dZ e e ed <d S)ExportBidsRequest.zPath to processed bids JSONrprocessed_bids_filerrjsonzExport format: json or csvr export_formatNz!Limit number of records to exportlimit) rr r!r r2r"r#rr4r5rintr$r$r$r%r17s r1c@sneZdZUedddZeeefed<edddZ eed<edddZ e e eed <ed d d Z eed <dS)ProcessGeMBidsDirectRequest.zRaw bids JSON datar bids_datarrNKeywords to filter sellersr)r*r+rr,)rr r!r r8rr"rr#rr)rrr,r$r$r$r%r7=s r7c@seZdZUedddZeed<edddZeed<edddZe e eed <ed d d Z eed <eddd Z eed<edddZ e eed<dS)ProcessGeMBidsFullRequest.rrrrrNr9r)r*r+rr,r3zr?r@ total_sellerstotal_bids_analyzed top_sellersanalysis_metadataN) rr r!r"r#rr6rrrr$r$r$r%rESs  rEc@s6eZdZUeed<eeefed<eeefed<dS)BidAnalyticsResponser>datasummaryN)rr r!r"r#rrr$r$r$r%rJ\s rJr process_typereturncCs"td||}|jddd|S)z$Get tenant-specific output directory/tmp/bid_processingTparentsexist_ok)rmkdir)rrMbase_dirr$r$r%get_tenant_output_diresrUr@cC tj|S)zVerify that a file exists)ospathexists)r@r$r$r%verify_file_existsk rZdir_pathcCrV)zVerify that a directory exists)rWrXisdir)r\r$r$r%verify_directory_existsor[r^root_dircCst|}d}d}d}|sdS|D]-}|rA|d7}tdd|D}tdd|D}|r=|r=|d7}q|d7}q|||fS)zz Count folders with required files (PDF and JSON) Returns: (total_folders, folders_with_files, missing_count) r)rrrcss&|]}|r|jdkVqdS)z.pdfN)is_filesuffixlower.0fr$r$r% s$z4count_folders_with_required_files..css"|] }|r|jdkVqdS)zstage_summary_data.jsonN)ranamerdr$r$r%rgs )rrYiterdiris_dirr*)r_ root_path total_folders valid_foldersinvalid_foldersfolderhas_pdfhas_jsonr$r$r%!count_folders_with_required_filesss    rrz"/tmp/bid_processing/.task_registryTrPcCs ttS)zGenerate unique task ID)r"uuiduuid4r$r$r$r%create_task_idr[rutask_idr.rcCsrt|d}||||tdd}t|d}t||Wdn1s*wYtd|dS)z*Save task metadata for later status lookup.json processing)rvrr.r created_atr>wNzTask metadata saved for ) TASK_REGISTRY_DIRr now isoformatopenr3dumploggerinfo)rvrr.r task_filemetadatarfr$r$r%save_task_metadatas  rcCsNt|d}|r%t|d }t|WdS1s wYdS)zRetrieve task metadatarwrN)r{rYr~r3load)rvrrfr$r$r%get_task_metadatas   rr> result_datacCst|d}|r_t|d }t|}Wdn1s wY||d<t|d<|r7||d<t|d}t||Wdn1sMwYt d|d |dSdS) zUpdate task status and resultrwrNr> completed_atresultrzTask z status updated to ) r{rYr~r3rr r|r}rrr)rvr>rrrfrr$r$r%update_task_statuss   rc Csz7td|t|}|std|WdSt|d|d|d}t|d|td|d WdStyg}z$tjd|d t|d d t|d t|t|j dWYd}~dSd}~ww)z-Wrapper to run handler and update task statusz(Starting background processing for task zTask metadata not found for Nr.rr completedrz completed successfullyz failed: Texc_infofailederror error_type) rrrrprocess_eproc_bids_handlerr Exceptionr"typer)rvrrer$r$r%%process_eproc_bids_background_wrappers* r input_file output_filec Cst}ztd|t|std|t|d}||}t|ddd }t|}Wdn1s8wYtdt | d gd t |}t|d dd}tj ||d d dWdn1slwYt|} tdd|d D} ddt |d dt|| t |d | | didd} td| dd| WSty} z(tjdt| ddddt| dt|ddt| idWYd} ~ Sd} ~ ww) zProcess GeM bid dataz'Starting GeM bid processing for tenant Input file not found: gem_processingrutf-8encodingNLoaded bidsz bids from input filerzFindent ensure_asciics8|]}t|didididgVqdS detailedInfosections evaluationsellersNlengetrebidr$r$r%rg & z+process_gem_bids_handler..successSuccessfully processed  bidsr)rFrr>r?r@rArBrCz GeM bid processing completed in .2fszError processing GeM bids: TrrzFailed to process GeM bids: r)timerrrZFileNotFoundErrorrUr~r3rrr gem_refineprocess_bids_datarsumr"rr) rrr start_time output_dir output_pathrfrKprocessed_datarArFrrr$r$r%process_gem_bids_handlersR          rr*r)r,c Cst}ztd|t|std|t|d}||}tj|t|||dt |ddd }t |} Wdn1sCwYt|} | d gdd } d d t | d gd t|| d ddtdd| d gD| | d d| d d| d d| dd} td| dd| WSty} z*tjdt| ddddt| dddgt| t|d dWYd} ~ Sd} ~ ww)!z#Analyze sellers from processed bidsz$Starting seller analysis for tenant rseller_analysisr)r,rrrNrrzSuccessfully analyzed  unique sellersrtotal_unique_sellersrcs|] }|ddVqdStotal_bids_appliedrNrrerr$r$r%rgGz*analyze_sellers_handler.. analysis_datesource_total_bidsfilter_applied)rrrrA)r>r?r@rFrGrHrIzSeller analysis completed in rrzError analyzing sellers: TrrzFailed to analyze sellers: )rrA)rrrrZrrU seller_statsanalyze_biddersr"r~r3rrrrrr)rrrr)r,rrrrfresultsrArHrrr$r$r%analyze_sellers_handler!sX       rcCst}ztd|td|t|s td|t|d}||}t|\}}}td|d|d|d|d krJtd ||d krRtd t |t |t d } td |d| } t|} t| dg} tdd| dgD} dd| d|dt || | | |||| di| didd | diddddd}td| ddtd | d!| d"|WSty}z'td#t |d$d%t |d&t|d t |d'd(dWYd&}~Sd&}~wty5}z'td)t |d$d*t |d&t|d t |d+d(dWYd&}~Sd&}~wtyj}z)tjd,t |d-d.d$d/t |d&t|d t |d0d(dWYd&}~Sd&}~ww)1ue Process eproc bid data from work order PDFs and JSON files Expected folder structure: root_directory/ ├── folder_1/ │ ├── work_order.pdf (or any .pdf file) │ └── stage_summary_data.json ├── folder_2/ │ ├── work_order.pdf │ └── stage_summary_data.json ... z)Starting eproc bid processing for tenant zRoot directory: zRoot directory not found: eproc_processingzFound z folders: z valid, z invalidrzNo folders found in z]No valid bid folders found. Each folder must contain a PDF file and 'stage_summary_data.json'ANTHROPIC_API_KEYzStarting to process z valid folders...rcsrrrrr$r$r%rgrz-process_eproc_bids_handler..rrz eproc bids from z foldersrwithDetailedInfodetailFetchSuccessz0%)bids_with_detailed_infodetail_fetch_success_rate)rFfolders_processedfolders_skippedtotal_folders_foundrprocessing_summaryrz"eproc bid processing completed in rrz Processed z bids with rz%Directory error in eproc processing: rzDirectory error: Ndirectory_not_foundrz&Validation error in eproc processing: zValidation error: validation_errorzError processing eproc bids: TrzFailed to process eproc bids: processing_error)rrrr^NotADirectoryErrorrUrr ValueErroreproc BidProcessorr"rWgetenvprocess_all_foldersrrrrr)r.rrrrrrlrmrn processorrrA total_bidsrF final_resultrr$r$r%r`s               r analysis_filec Csztd|t|std|t|ddd }t|}Wdn1s+wY|dg}|di}td d |D}td d |D}t ||td d |Dtd d |D||dd|rp|t |nd|dkr{||dnd|ddt t dd|Dt dd|Dt dd|Ddd }d||t dd} | WSty} z!tjdt| dddit| t d dWYd} ~ Sd} ~ ww)!$Get analytics from bid analysis dataz$Generating bid analytics for tenant zAnalysis file not found: rrrNrrcsrrrrr$r$r%rgrz,get_bid_analytics_handler..csrtotal_bids_won_l1rNrrr$r$r%rgrcsrtotal_bids_qualifiedrNrrr$r$r%rgrcsrtotal_bids_disqualifiedrNrrr$r$r%rgrtotal_amount_won_all_l1_bidsu₹0.00rd cSs"g|] }|dgD]}|q qS)seller_locationsr)rerlocr$r$r% s"z-get_bid_analytics_handler..cS g|] }d|ddvr|qS)MSEspecial_statusrrr$r$r%r cSr)MIIrrrrr$r$r%rr)msemii) rFrtotal_qualifiedtotal_disqualifiedtotal_l1_winnerstotal_amount_wonaverage_bids_per_seller success_ratetop_10_sellersrspecial_categoriesr)r generated_at)r>rKrLzError generating analytics: Trr)rr)rrrZrr~r3rrrrlistsetr r|r}rrr") rrrf analysis_datarrrr analyticsrrr$r$r%get_bid_analytics_handlersP       r z/process/gem-bids)response_modelcCtjddSNBID_PROCESSING_API_KEYz default-keyrWenvironrr$r$r$r% rrequestbackground_tasksapi_keyc szzt|j|j|j}|ddkrtd|ddtdi|WSty<}ztd|tdt |dd}~ww) z"Process GeM bid data and refine itr>rr? status_codedetailz$Error in process_gem_bids endpoint: Nr$) rrrrrr=rrrr")rrrrrr$r$r%process_gem_bids s rz/process/eproc-bidscCrrrr$r$r$r%r(rc sz=td|jt}td|d|jt||j|j|j|t|t dd|dddd|d |d d WSt y\}ztj d |d dt dt |dd}~ww)z Process eproc bid data from work order PDFs and JSON files (Non-blocking) Returns task_id that can be used to check status via GET /process/eproc-bids/status/{task_id} z.Received eproc processing request for tenant: z Created task z for tenant rxz$eproc processing started - task ID: Nrprocessing_in_backgroundz./api/bid-processing/process/eproc-bids/status/)r>rvpoll_urlrz#Error initiating eproc processing: Trrr)rrrrurr.radd_taskrr=rrrr")rrrrvrr$r$r%process_eproc_bids$s>  r!z$/process/eproc-bids/status/{task_id}c sztd|t|}|s#td|d|dd|dWS|ddkrNt|d }t|d }d||d |d t |d d ddWS|ddkr|di}d||d |d |d|d|d|d|d|dd WS|ddkr|di}d||d |d |d|d|dd|ddWSd||dd|ddWSt y}ztj d |d!d"d|t |d#dWYd$}~Sd$}~ww)%z Check processing status using task_id Returns: - processing: Still running - completed: Finished successfully with results - failed: Error occurred - error: Task not found or invalid zChecking status for task zTask not found: rzTask not foundzNo task found with ID: )r>rvrr?r>rxry<rr`z5Processing in progress. Check again in a few minutes.z#5-15 minutes depending on file size)r>rvrryelapsed_minutesr?estimated_waitrrrrBr@rAr?rC) r>rvrryrrBr@rAr?rCrrzProcessing failed: )r>rvrryrrrr?unknownzUnknown status: )r>rvmetadata_statusr?zError checking task status: TrzError checking task statusN) rrrwarningr fromisoformatr| total_secondsrroundrrr")rvrelapsedr#rrr$r$r%get_eproc_statusZsx      r,z/analyze/sellerscCrrrr$r$r$r%rrc sz$t|j|j|j|j|jd}|ddkrtd|ddtd i|WStyA}zt d|tdt |dd}~ww) z'Analyze sellers from processed bid datarr>rrr?rz#Error in analyze_sellers endpoint: Nr$) rrrrr)r,rrErrrr"rrrrr$r$r%analyze_sellerss" r.z /analyze/bidscCrrrr$r$r$r%rrc s~z!t|j|j}|ddkrtd|ddddtd i|WSty>}ztd|tdt |dd }~ww) rr>rrrLrz Unknown errorrz%Error in get_bid_analytics endpoint: Nr$) r r0rrrrJrrrr"r-r$r$r%get_bid_analyticss r/z/processing-status/{tenant_id}cCrrrr$r$r$r%rrc szGtd|}|s|dddWSi}|D](}|rAt|d}t|dd|D|r:tdd|Dnd d ||j<q|d |d WSt yd}zt d |t dt |dd }~ww)z*Get status of processing jobs for a tenantrO no_processingz+No processing history found for this tenant)rr>r?z*.jsoncSsg|]}|jqSr$)rhrdr$r$r%r#sz)get_processing_status..cSs g|] }t|jqSr$)r fromtimestampstatst_mtimer}rdr$r$r%r$rN) file_countfiles last_modifiedr)rr>processing_historyz!Error getting processing status: rr)rrYrirjr globrmaxrhrrrrr")rr output_baseprocessing_dirsrMr5rr$r$r%get_processing_status s4    r<z"/export/{tenant_id}/{process_type}cCrrrr$r$r$r%r6r file_namec sz:t||}||}|stdd|dt|ddd }t|}Wdn1s/wYd|||d WStyCty^}ztd |td t |dd}~ww) zExport processed bid dataizFile not found: rrrrNr)r>rr=rKzError exporting data: r) rUrYrr~r3rrrrr") rrMr=rrr@rfrKrr$r$r%export_processed_data1s*   r>z/process-and-analyzecCrrrr$r$r$r%rTrc st}zNtd|j|jrt|jtstdddtd|j}d|vr.csrrrrr$r$r%rgrr bids and analyzed  sellersrcsrrrrr$r$r%rgr)r>bids_processedtotal_sellers_foundcsrrrrr$r$r%rgrcsrrrrr$r$r%rgrrr)r>unique_sellersrGrrrr)step_1_gem_processingstep_2_seller_analysisr) processed_atkeywords_filteredr,) r>rrAr?rprocessed_bids_dataseller_analysis_datarrz.Combined processing completed successfully in rrz'Error in process_and_analyze_gem_bids: Trr$Failed to process and analyze bids: )rrrrr8 isinstancedictrr r|r}rrrrrUr6r~r3rrrr"r)r,rrrr)rrrdata_to_processrrprocessed_file_pathrfr0analysis_resultsrArrrresponserr$r$r%process_and_analyze_gem_bidsQs                " rYz/process-and-analyze-filecCrrrr$r$r$r%rrc sNt}ztd|jtd|j|jdvr$tdddt|js3tdd|jdtd|jt|jd d d  }t |}Wd n1sTwYtd t | dgdtdd|vrt t | dgd|d<t|}tdt |ddt|jd}tt}|d|d}|d|d} t|dd d }t j||dddWd n1swYtdtjt|t| |j|jdt| d d d  }t |} Wd n1swYt|} | dg} td d!| D} td"d!| D}| t |dt | | td#d!| Dtd$d!| D|| d%krD|| d&nd%d'}z || Wntym}ztd(|WYd }~nd }~ww|jd)krtd*| }||d+<d,|d-<td.| d/d0|WS|jrt |j}|j!j"d1d1d2n|d3}td4|t|dd d }t j| |dddWd n 1swYtd5| d/d0d,dt |dd6t | d7t|| t | t||d8d9WStyty&}ztj#d:t|d1d;tda Combined endpoint: Process GeM bids from file and analyze sellers in ONE call Parameters: - input_file_path: Path to input JSON file - tenant_id: Tenant identifier - search_keywords: Optional keywords to filter sellers - search_mode: 'any' or 'all' for keyword filtering - return_type: 'json' (return in response body) or 'file' (save to disk) - output_file_path: Optional custom output file path Returns ONLY the final seller analysis (no intermediate files) r?z Return type: )r3filer@z$return_type must be 'json' or 'file'rrz Step 1: Loading bids from file: rrrNrrz bids from filezStep 2: Processing GeM bids...rrArrrDz.temp_processed_rwz.temp_analysis_rzrFrzStep 3: Analyzing sellers...rrcsrrrrr$r$r%rgrz4process_and_analyze_gem_bids_file..csrrrrr$r$r%rgrcsrrrrr$r$r%rgrcsrrrrr$r$r%rgrrr)rArIrKrGrrrrzCould not clean up temp files: r3z2Returning seller analysis as JSON in response bodyrrr>zProcessing completed in rrTrPr(zSaving seller analysis to: z"Processing completed and saved in rGrH)rstatsrz,Error in process_and_analyze_gem_bids_file: rrrR)$rrrrr;rrZrr~r3rrrr r|r}rrrUr6rrrr"r)r,runlinkrr'copyr<rparentrSr)rrrrfr8rr timestamptemp_processed_filetemp_analysis_filerWrArrr summary_datar response_datar<r$r$r%!process_and_analyze_gem_bids_files                    rd)N)Nr*)J__doc__rWr3loggingtypingrrrrfastapirrrr r pydanticr r rrsr pathlibrrefine_gem_bid_datarrrprocess_eproc_datar basicConfigINFO getLoggerrrrouterrr'r-r/r1r7r:r=rErJr"rUboolrZr^tuplerrr{rSrurrrrrrrr postrr!rr,r.r/r<r>rYrdr$r$r$r%s     " >   "?o :    5       $    w