o if @sjdZddlZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl Z ddlZddlZddlmZddlZddlmZddlmZddlmZddlZddlZejejdd ee Z!zej"d d d ej"d d d WnYdZ#GdddZ$de%de$fddZ&de%de%de%de e'e%ffddZ(dde%de%de%de e%e%ffddZ)dS)z Unified Document Text Extraction Module This module provides comprehensive text extraction capabilities for tender RFP documents. It handles PDF, Excel, Word documents and scanned images, returning both individual document texts and merged content. N)datetime)DictTupleListOptional) docx2python) word_tokenize) content_typesz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformatpunktT)quiet punkt_tabclaudec @s eZdZdZd,defddZddZdd Zd d Zd ed e fddZ d e fddZ d-dede d efddZ d.deded efddZdeded efddZd,d ed!eed eeeefe effd"d#Zd$eeefd%ed e fd&d'Zd$eeefd e fd(d)Zd*d+ZdS)/DocumentExtractorz Unified document text extraction class that handles multiple file types and provides both individual and merged text outputs. N llm_api_keycCsT||_||_tdkr|rtj|dnd|_ntdkr%|r"tj|dnd|_i|_ dS)z Initialize the document extractor. Args: llm_api_key (str): API key for Gemini (used for scanned documents) process_logger: Optional process logger for detailed logging gemini)api_keyNr) rprocess_logger llm_modelgenaiClient gemini_client anthropic Anthropic claude_client_extraction_cache)selfrrr=/var/www/html/minaions-tender/ai-engine/document_extractor.py__init__0s zDocumentExtractor.__init__cC$t||jr|j|dSdS)z9Log info message to both module logger and process loggerN)loggerinforrmsgrrr _log_info@ zDocumentExtractor._log_infocCr!)zsz;DocumentExtractor.extract_text_from_pdf..z, cSsg|]}t|dqSr=strr>rrrrAsz/ This message contains a> scanned pages from a PDF document. For EACH PAGE, extract ALL text content visible in it. Format your response with clear page number headers like: "=== NEW PAGE ===" [extracted text for page 1] "=== NEW PAGE ===" [extracted text for page 2] And so on for each page. Format the text naturally, preserving paragraphs, bullet points, and other structural elements. Please just give out the extracted text as output without any additional commentary. NOTE that the text in the scanned pages can be in any language. rzgemini-2.5-flash-preview-04-17zapplication/pdf)data mime_type)modelcontentsrutf-8zclaude-3-5-haiku-latestig?zYou are an expert at extracting text of any language from scanned documents. Extract ALL text visible on EACH page, preserving the original formatting as much as possible.userr,)typer,documentbase64)rJ media_typerD)rJsource)rolecontent)rF max_tokens temperaturesystemmessagesz=== NEW PAGE ===Nz" --- [No text content extracted] z' --- [Could not extract page content] zError extracting text from PDF : zError processing PDF: zProcessing complete. Processed z pages, including z scanned pages.),openPyPDF2 PdfReaderr.pagesr&ospathbasenameranger5appendr2 PdfWriteradd_pageioBytesIOwriteseekjoinrgetvaluermodelsgenerate_contenttypesPart from_bytesr,rL b64encodereaddecoderrTcreaterPr/ enumerater3timesleep ExceptionrC)rr7r8extracted_text scanned_pagesfile pdf_reader total_pagespage_numr4r,ibatchwriter output_buffer page_listpromptbatch_pdf_bytesresponse llm_text_out pdf_base64scanned_pages_textj page_contente combined_textrrrextract_text_from_pdfgs        $           z'DocumentExtractor.extract_text_from_pdf || excel_filepath separatorc szOtj|dddtfddjD}|ddtj|D}g}D]\}}|ddt||D}| |q0d |g|WSt yu} z| d |d t| d t| WYd } ~ Sd } ~ ww)ai Converts an Excel file to a formatted text file, maintaining table structure. Args: excel_filepath (str): Path to the Excel file. separator (str, optional): Separator character between columns. Defaults to " || ". Returns: str: Formatted text representation of the Excel file r)headerrVcs*g|]}tdtdd|jDqS)2css|] }tt|VqdSN)r.rC)r?xrrr szHDocumentExtractor.extract_table_from_excel...)minmaxvalues)r?coldfrrrAs*z>DocumentExtractor.extract_table_from_excel..css"|] \}}|d|VqdS.css&|]\}}t|d|VqdSrrB)r?cellrrrrr$s  zError extracting from Excel rUzError processing Excel: N) pd read_excelfillnaastyperCcolumnsrfzipiterrowsr_rtr+) rrr col_widths header_line data_lines_row data_linerrrrextract_table_from_excel s(       z*DocumentExtractor.extract_table_from_excel word_pathtemp_dirc Csz)tjtj|d}tj||d}||}tj|r't||WStyO}z| d|dt |dt |WYd}~Sd}~ww)a# Extract text from Word document by converting to PDF first. Args: word_path (str): Path to the Word document temp_dir (str): Directory to store temporary PDF file Returns: str: Extracted text content r.pdfzError extracting from Word rUz Error processing Word document: N) r[r\splitextr]rfget_structured_docx_contentexistsremovertr+rC)rrr base_name pdf_filenamer,rrrrextract_text_from_word.s    z(DocumentExtractor.extract_text_from_word input_dir cache_keyc Cs|r||jvr|d||j|Stj|s#idd|dfSg}g}g}t|D])}tj||}|drB||q.|drM||q.|drW||q.|sf|sf|sfidd|fS|d t |d t |d t |d i}g} g} t } |D]Y}z5| |} | r| d s| ||<| ||dtj|n| ||dtj|Wqty} z|d|d| | |WYd} ~ qd} ~ ww|D]\}z7||} | r | d s | ||<| ||dtj|n| ||dtj|Wqty?} z|d|d| | |WYd} ~ qd} ~ ww|D]_}z9|||} | rj| d sj| ||<| ||dtj|n| ||dtj|WqBty} z|d|d| | |WYd} ~ qBd} ~ wwt | }|siddf}ntdd|D}|ddt | d|d|f}|r||j|<|S)a Extract text from all supported documents in a directory. Args: input_dir (str): Directory containing documents to process cache_key (str, optional): Key for caching results Returns: Tuple[Dict[str, str], bool, str]: - Dictionary mapping file paths to extracted text - Success status - Status message z$Using cached extraction results for Fz Directory z does not exist)z.xlsz.xlsxz.XLSz.XLSX)rz.PDF)z.docz.docxz.DOCz.DOCXz%No supported document files found in r<z PDF files, z Excel files, z Word files to processErrorz!Successfully extracted text from zFailed to extract text from zError processing PDF rUNzError processing Excel zError processing Word z5No text was successfully extracted from any documentscss|]}t|VqdSr)r.)r?r,rrrrsz;DocumentExtractor.extract_documents_text..Tz documents (z total characters) in )rr&r[r\rlistdirrfendswithr_r.rnowr startswithr]r)rtr+rrsumr)rrr pdf_files excel_files word_filesfilename file_pathdocuments_textsuccessful_files failed_files start_timer,rprocessing_timeresult total_charsrrrextract_documents_textKs         *             z(DocumentExtractor.extract_documents_textr output_pathc Csz>d|}tjtj|ddt|ddd }||Wdn1s*wY|dt |d |WdSt yY}z| d |WYd}~d Sd}~ww) aC Create a merged text file from individual document texts. Args: documents_text (Dict[str, str]): Dictionary mapping file paths to text content output_path (str): Path where to save the merged text file Returns: bool: Success status r;T)exist_okwrHencodingNz Merged text (z chars) successfully written to z!Error creating merged text file: F) rfrr[makedirsr\dirnamerWrdr&r.rtr+)rrr merged_textrwrrrrcreate_merged_text_files  z)DocumentExtractor.create_merged_text_filecCsfd}|D]!\}}|s q||}||7}|dtj|d|ddq|d|d|S)a Count the total number of words in all document texts. Args: documents_text (Dict[str, str]): Dictionary mapping file paths to their text content Returns: int: Total word count across all documents rz Document: z - ,z wordsz# Total words across all documents: )itemsr0r&r[r\r])rrtotal_word_countr text_content word_countrrrcount_total_wordss  $z#DocumentExtractor.count_total_wordsc Csjzt|}t|jWdWS1swYWdSty4}z d|WYd}~Sd}~ww)z9Returns the content of a .docx file as a structured list.NzError reading file: )rrCbodyrt)rr docx_contentrrrrrs (z-DocumentExtractor.get_structured_docx_contentr)r6)r)__name__ __module__ __qualname____doc__rCr r&r)r+intr0boolr5rrrrrrrrrrrrrrr*s %#.i rrr-cCst|S)z Factory function to create a DocumentExtractor instance. Args: llm_api_key (str): API key for Gemini Returns: DocumentExtractor: Initialized extractor instance )r)rrrrcreate_document_extractors rr output_dirc Cstj|d}tj|rdd|dfSt|}|j||d\}}}|s)d|fStj|d}t|dd d } tj|| dd d Wd n1sKwY| ||} | rZdSdS)aD Backward compatible version of extract_RFP_text for bid_queries.py Args: input_dir (str): Directory containing input documents output_dir (str): Directory for output files llm_api_key (str): API key for LLM Returns: Tuple[bool, str]: Success status and message merged.txtT The file '' already exists.rF doc_text.jsonrrHr ensure_asciiindentN)T%Successfully created merged text file)F!Failed to create merged text file) r[r\rfrrrrWjsondumpr) rrrmerged_file_path extractorrsuccessmessagedocuments_text_pathf merge_successrrrextract_RFP_text_compatibles  rcsfdd}fdd}fdd}tj|d}tj|rs|d|d z)t|d d d  }t|} Wd n1s?wY|dt| d| WStya|d|Yd Stj yr|d|Yd Sw|d|t |d} |d| j ||d\} } } |d|t|dd d } tj | | dddWd n1swY| r|d| |dt| dtj|d}|d|| | |}|r|d| S|d | S|d!| iS)"aH Backward compatible function for tender_automation.py Args: input_dir (str): Directory containing documents llm_api_key (str): API key for LLM process_logger: Optional process logger for detailed logging Returns: Dict[str, str]: Dictionary mapping file paths to extracted text c t|r|dSdSr)r"r#r%rrrlog_info4 z3extract_documents_text_compatible..log_infocrr)r"r*rrrr log_error9rz4extract_documents_text_compatible..log_errorcrr)r"r(rrrr log_warning>rz6extract_documents_text_compatible..log_warningrrrrrHrNz*Loaded existing document text from cache: z documentszError: File not found: zError: Invalid JSON format in: z"Starting document extraction from rz!Extracting text from documents...rzSaving extracted text to rFrrz Document extraction successful: zExtracted text from rzCreating merged text file at rrzDocument extraction failed: )r[r\rfrrWrloadr.FileNotFoundErrorJSONDecodeErrorrrrr)rrrrrrrrrwrrrrrrrrrr!extract_documents_text_compatible'sP       rr)*rr[rbsysrrLpathlibrtypingrrrrrrrXpandasrrnltk nltk.tokenizergoogle.generativeai generativeairgoogle.generativeai.typesr rjrlogging basicConfigINFO getLoggerrr"downloadrrrCrrrrrrrrsD      I"()