o gif @s.dZddlZddlZddlZddlZddlZddlmZmZm Z m Z ddl m Z m Z mZddlmZmZddlZddlmZddlmZddlmZejejd eeZe d d gd Zd@dedefddZeZ e !di!de"ddZ#e !di!de"dZ$GdddZ%GdddeZ&GdddeZ'dAde(defd d!Z)d"ede efd#d$Z*d%e%d&ed'ede efd(d)Z+d%e%d*ed'ede efd+d,Z, dAd"ed%e%d-ed.e(deee ff d/d0Z- 1   2dBd3e(d.e(de(d4e.deee ff d5d6Z/ej0d7e'd8d9e&d:efd;d<Z1e!d=d>d?Z2dS)Cz Location Finder API for Minaions Tender System Extracts delivery locations from GeM tender PDFs and updates backend To be imported and included in main app.py N)DictListOptionalAny) APIRouter HTTPExceptionBackgroundTasks) BaseModelField)datetime)Path)TenderLocationExtractor)levelz/api/location-finderzLocation Finder)prefixtags config.json config_pathreturnc Csxzt|d}t|WdWS1swYWdSty;}ztd|diWYd}~Sd}~ww)z#Load configuration from config.jsonrNzFailed to load configuration: z. Using environment variables.)openjsonload Exceptionloggerwarning)rfer>/var/www/html/minaions-tender/ai-engine/location_finder_api.py load_configs (rnodejs_serviceapi_urlBACKEND_API_URLz"http://localhost:5000/internal-apiapi_keyINTERNAL_API_KEYc@seZdZdZddZddededeefd d Zddededeefd d Z d e dedefddZ de de de fddZ de de de fddZdejddfddZdS)BackendApiClientz(Handles API calls to the Node.js backendcCs"|d|_||_|dd|_dS)N/zapplication/json)x-internal-api-keyz Content-Type)rstripbase_urlr#headers)selfr)r#rrr__init__5s  zBackendApiClient.__init__rlimitskiprcC>|jd}||d}tj||j|d}|||dS)zGet all tendersz/tendersr.r/r*paramsdatar)requestsgetr*_check_responserr+r.r/urlr3responserrrget_all_tenders=   z BackendApiClient.get_all_tenderscCr0)z0Get tenders that don't have location informationz/tenders/without-locationr1r2r4r5r9rrrget_tenders_without_locationHr=z-BackendApiClient.get_tenders_without_location tender_id update_datacCs8|jd|}tj|||jd}|||dS)z'Update tender with location informationz /tenders/)rr*r4)r)r6putr*r8r)r+r?r@r:r;rrr update_tenderSs  zBackendApiClient.update_tenders3_key tenant_idcCs>|jd}||dd}tj|d|ji|d}|||jS)z2Download file from S3 via backend API using S3 keyz/storage/downloadtrue)rDkeydecryptr'r2r)r6r7r#r8content)r+rCrDr:r3r;rrr download_fileZs  zBackendApiClient.download_file document_idcCs@|jd|d}d|i}tj|d|ji|d}|||jS)z;Download document content via backend API using document IDz /documents/z/contentrDr'r2rH)r+rKrDr:r3r;rrrdownload_document_by_idfs  z(BackendApiClient.download_document_by_idr;NcCsr|jdkr7d|j}z|}d|vr"|d|d}Wt|Wt||d|j}Yt|dS)z2Check response status and raise exception if erroriz API Error message: N) status_codertextr)r+r; error_msg error_datarrrr8ps  z BackendApiClient._check_response)r-r)__name__ __module__ __qualname____doc__r,intrrr<r>strrBbytesrJrLr6Responser8rrrrr%2s    r%c@s^eZdZUedddZeed<edddZeed<edddZeed <ed d dZ e ed <d S)ProcessAllTendersRequestTz*Only process tenders without location data) description only_missingFz!Use regex fallback instead of LLM use_fallbackz&Use Anthropic Claude (default: OpenAI) use_anthropicdz$Maximum number of tenders to processr.N) rSrTrUr r]bool__annotations__r^r_r.rWrrrrr[s r[c@seZdZUeed<eed<dZeeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dZ ee ed <gZeeeeefed <d S) ProcessAllTendersResponsestatusrMr total_tenders processed successfulfailedskippedprocessing_timeresultsN)rSrTrUrXrbrerrWrfrgrhrirjfloatrkrrrrrrrrcs  rcFr_cCs8|rtd}|s td|Std}|std|S)z>Get the appropriate API key from environment based on providerANTHROPIC_API_KEYzDANTHROPIC_API_KEY not found in environment. Set it to use Anthropic.OPENAI_API_KEYz>OPENAI_API_KEY not found in environment. Set it to use OpenAI.)osgetenv ValueError)r_r#rrr get_api_keys  rrtendercCsd|vrT|drTt|dtrTt|ddkrT|dd}tdt|dd|t|trE|dp:|d}td||St|trTtd||Sd |vrftd |d |d Sd|vrxtd |d|dSd |vr|d rt|d trt|d dkr|d d}tdt|d d |t|tr|dp|d}td||St|trtd||St ddS)z Extract the FIRST document ID/URL/S3 key from tender data. Priority: originalDocuments[0] > documentUrl > s3Key > documents[0] Returns: Document ID (24-char hex), S3 key, or URL of the first tender document originalDocumentsrzFound z documents, using FIRST: r:s3Keyz,Extracted from originalDocuments[0] (dict): z.Extracted from originalDocuments[0] (string): documentUrlzExtracted from documentUrl: zExtracted from s3Key: documentsz" in documents array, using FIRST: zExtracted from documents[0]: z*No document reference found in tender dataN) isinstancelistlenrdebugdictr7inforXr)rs first_docresultrrrextract_pdf_url_from_tenders<        r api_clientrKrDc CsHd}zmtd||||}|rt|dkr$td|WdS|ds@td|td|ddWdStjd d d }|||j }Wdn1sZwYtd |d t|d|WSt y}z(td|dt ||rt j |rzt |WnYWYd}~dSd}~ww)zADownload PDF using document ID and save to temp file, return pathNz!Downloading PDF for document ID: rz*Downloaded file is empty for document ID: %PDF-zIDownloaded file is not a valid PDF (missing PDF header) for document ID: File starts with: F.pdfdeletesuffixDownloaded PDF to:  ( bytes)z&Error downloading PDF for document ID rN)rr}rLrzerror startswithtempfileNamedTemporaryFilewritenamerrXropathexistsunlink)rrKrD temp_path file_content temp_filerrrrdownload_pdf_by_document_ids6   rrCc CsBd}zmtd||||}|rt|dkr$td|WdS|ds@td|td|ddWdStjd d d }|||j }Wdn1sZwYtd |d t|d|WSt y}z%tdt ||rt j |rzt |WnYWYd}~dSd}~ww)zDDownload PDF from S3 using S3 key and save to temp file, return pathNzDownloading PDF from S3: rz%Downloaded file is empty for S3 key: rzDDownloaded file is not a valid PDF (missing PDF header) for S3 key: rrFrrrrrzError downloading PDF from S3: )rr}rJrzrrrrrrrrXrorrr)rrCrDrrrrrrrdownload_pdf_from_s3s6   r extractorr^c Cst|dd}t|dd}|dd}|||dddd}z|s3|d r3d |d <d |d <|WSt|}|sd|d <td|d|dddddtddddddtdddddid} z| || d|d <ddd|d<W|WSt y} zt d t| WYd} ~ |WSd} ~ wwt |tot |d!kotd"d#|Dod$|vod%|v} | rtd&|t|||} ntd'|t|||} | sSd(| rd)nd*d+|d,|d <td|d|d-| rd)nd*d+|d. ddddtddddddtddd/ddid} z| || d|d <ddd|d<W|WSt yR} zt d t| WYd} ~ |WSd} ~ wwztd0|d|d1|j| |d2} | r| d3r| d4| d3| d5| d6d7t| d8d9d| d3| d3ddtd:| d3| d4| d6d;did} | || d|d <| |d<td<|d=| d3n=ddddtddddddtddd>ddid} | || d|d <ddd|d<d?|d <td@|dAW| rtj| r t| W|SW|SW|S| r3tj| r4t| wwwt y]} zt| |d <tj dB|d+t| dCdDWYd} ~ |Sd} ~ ww)Ez6Process a single tender to extract and update location_idtenant bidNumberUnknownrhN)r? bid_numberrDrdlocationrdeliveryLocationrirdzLocation already existsrz/No PDF document found in tender, set to "Other"zTender rz5): No PDF document found, setting location to 'Other'Otherlowfallback)citystate fullAddress confidence extractedAtextractionMethod)zmetadata.location_infozmetadata.locationrlogslocation_extraction_failedzNo PDF document found)rreason) timestampactiondetails)z$setz$pushsuccess)rrrz/Failed to update tender with 'Other' location: css|]}|dvVqdS)0123456789abcdefABCDEFNr).0crrr jsz(process_single_tender..r&.zDetected document ID: zDetected S3 key: zFailed to download PDF (z document IDzS3 keyrNz), set to "Other"z): Failed to download PDF from z, setting location to 'Other'zFailed to download PDFProcessing tender ))r^rr full_addressrmediummethodllmlocation_extracted)rrrzUpdated tender z with location: zNo valid location extractedz+No valid location extracted, set to "Other"z'No valid location extracted for tender z, set location to 'Other'zError processing tender Texc_info)rXr7rrrr now isoformattimerBrrrxrzallr}rrprocess_tender_pdfrorrr)rsrrr^r?rDrr pdf_referencer@ris_document_idpdf_path location_datarrrprocess_single_tender sF         ,              (rTr`r]r.c CsFt}ztd|d|dttt}t|d}t||d}|rlz|j|d}tdt |dWnFt yk} z)t d t | d |j |d}d d |D}td t |dWYd} ~ nd} ~ ww|j |d}tdt |d|d|}g} d} d} d} t|D]9\}}td|ddt |t||||}| ||ddkr| d7} q|ddkr| d7} q| d7} qt|}ddt |d| d| d| d t |t || | | || d WSt y"} z'tjdt | dd d!d"t | dddddt|gd WYd} ~ Sd} ~ ww)#zD Main function to process all tenders and extract locations z7Starting location extraction for tenders (only_missing=, limit=r)r_)r#r_)r.zFetched z tenders without locationzSpecialized endpoint failed: z, fetching all tenderscSsg|] }|ds|qS)r)r7)rtrrr sz/process_all_tenders_handler..z Filtered to Nz tendersrrr&rdrri completedz Processed z tenders: z successful, z failed, z skipped rdrMrerfrgrhrirjrkz&Error in process_all_tenders_handler: TrrzProcessing failed: )rrr}r%r"r$rrr r>rzrrrXr< enumeraterappendr)r]r^r_r. start_timerr#rtendersrrkrgrhriidxrsrrjrrrprocess_all_tenders_handlerst      "             rz/process-all-tenders)response_modelrequestbackground_tasksc sz3td|jd|jtd|jt|j|j|j|jdtdd|jdddddddgd WSt yR}ztj d |d d t d t |dd}~ww)a Process all tenders to extract location information (NON-BLOCKING) This endpoint: 1. Starts processing in the background immediately 2. Returns a response indicating processing has started 3. Processing happens asynchronously without blocking other API requests Background processing: 1. Fetches all tenders from the backend 2. For each tender, downloads the PDF document from S3 3. Extracts location information using LLM (city, state, full address) 4. Updates the tender record in the backend with the extracted location Parameters: - only_missing: Default True - Only process tenders without location data - use_fallback: Default False - Use regex fallback instead of LLM - use_anthropic: Default False - Use Anthropic Claude instead of OpenAI - limit: Default 100 - Maximum number of tenders to process in one run This endpoint is designed to be run as a cron job once a day. z3Received process-all-tenders request: only_missing=rz=Starting background task for tender processing (non-blocking))r]r^r_r. processingz4Location extraction started in background for up to z2 tenders. Processing will continue asynchronously.rrz Error starting background task: Tri)rOdetailN) rr}r]r.add_taskrr^r_rcrrrrX)rrrrrrprocess_all_tendersNs6    rz/healthcsdddtdS)z-Health check endpoint for location finder APIhealthyzlocation-finderz2.0.0)rdserviceversion backend_api)r"rrrr health_checks r)r)F)TFFr`)3rVrorloggingr6rtypingrrrrfastapirrrpydanticr r rr pathlibr location_finder_gemr basicConfigINFO getLoggerrSrrouterrXrconfigr7rpr"r$r%r[rcrarrrrrrrWrpostrrrrrrsv     N 3&*  W  Y5