o [2iX@s8dZddlZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl Z ddlZddlZddlmZddlZddlmZddlmZddlmZddlZddlZddlmZddl Z ej!ej"d d e#e$Z%zej&d d d ej&dd d WnYdZ'GdddZ(de)de(fddZ*dS)z Enhanced Document Text Extraction Module Extends the original document_extractor.py to handle additional file types including CSV, images, and text files, with recursive folder processing. N)datetime)DictTupleListOptional)convert) word_tokenize) content_types)Imagez4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformatpunktT)quiet punkt_tabclaudec @s(eZdZdZdefddZdedefddZdefd d Z d.d ed edefddZ d/dededefddZ dedefddZ dedefddZ dedefddZdededefd d!Zd"edefd#d$Zd0d"ededefd&d'Zd(edeefd)d*Zd0d(ed+eedeeeefeeffd,d-Zd%S)1EnhancedDocumentExtractorz Enhanced document text extraction class that handles multiple file types and provides recursive folder processing capabilities. llm_api_keycCst||_tdkr|rtj|dnd|_ntdkr"|rtj|dnd|_i|_dgddgdd gd gd ggd d |_ dS)z Initialize the enhanced document extractor. Args: llm_api_key (str): API key for Claude/Gemini (used for scanned documents and images) gemini)api_keyNr.pdfz.xlsz.xlsxz.docz.docxz.csvz.txtz.jpgz.jpegz.pngz.tiffz.tifz.bmp)pdfexcelwordcsvtextimage) r llm_modelgenaiClient gemini_client anthropic Anthropic claude_client_extraction_cachesupported_extensions)selfrr'F/var/www/html/minaions-tender/ai-engine/enhanced_document_extractor.py__init__0s z"EnhancedDocumentExtractor.__init__rreturncCs&ztt|WSt|YS)zCount words in text.)lenrsplit)r&rr'r'r( count_wordsHsz%EnhancedDocumentExtractor.count_wordscCs|}| pt|dkS)z Determine if a PDF page is a scanned image using PyPDF2 Args: page: PyPDF2 page object Returns: bool: True if the page appears to be scanned, False otherwise ) extract_textr+strip)r&pagerr'r'r(is_scanned_pageOs z)EnhancedDocumentExtractor.is_scanned_pagepdf_path batch_sizecCs6i}g}zIt|d8}t|}t|j}tdtj |d|dt |D].}|j|} | | rH| |td|ddq+| } d|dd | d ||<q+td t|d |r=|jr=t d t||D]} || | |} | sqttdt| ddd| Dt} | D] }| |j|qt}| ||d dt| d}t|d}|jjjdddddd|dddd|dd gd!gd"}|jd j}|d#}|rt|t| kr|dd$}t| D]7\}}|t|kr,||}|r!d|dd |d ||<qd|dd%||<qd|dd&||<qt !dqtWd$n 1sHwYWn't"yu}ztd'|d(t#|d)t#|WYd$}~Sd$}~wwd*}t |D]}||vr|||7}q|td+|d,t|d-|S).a Extract text content from PDF files, using Claude for scanned pages in batches and PyPDF2 for pages with selectable text Args: pdf_path (str): Path to the PDF file batch_size (int): Maximum number of scanned pages to send to Claude in one request Returns: str: Extracted text content rbzProcessing PDF z with z pageszPage z appears to be a scanned page.z--- Doc Page Number:  ---  Found z scanned pages in the documentrzProcessing batch of z scanned pages: cSsg|]}|dqS)r7r').0pr'r'r( szCEnhancedDocumentExtractor.extract_text_from_pdf..z/ This message contains a> scanned pages from a PDF document. For EACH PAGE, extract ALL text content visible in it. Format your response with clear page number headers like: "=== NEW PAGE ===" [extracted text for page 1] "=== NEW PAGE ===" [extracted text for page 2] And so on for each page. Format the text naturally, preserving paragraphs, bullet points, and other structural elements. Please just give out the extracted text as output without any additional commentary. NOTE that the text in the scanned pages can be in any language. utf-8claude-haiku-4-5皙?zYou are an expert at extracting text of any language from scanned documents. Extract ALL text visible on EACH page, preserving the original formatting as much as possible.userrtyperdocumentbase64zapplication/pdfrD media_typedatarDsourcerolecontentmodel max_tokens temperaturesystemmessagesz=== NEW PAGE ===Nz" --- [No text content extracted] z' --- [Could not extract page content] zError extracting text from PDF : zError processing PDF: zProcessing complete. Processed z pages, including z scanned pages.)$openPyPDF2 PdfReaderr+pagesloggerinfoospathbasenameranger2appendr/r# PdfWriteradd_pageioBytesIOwriteseekrF b64encodereaddecoderTcreaterNrr, enumerater0timesleep Exceptionstr)r&r4r5extracted_text scanned_pagesfile pdf_reader total_pagespage_numr1ribatchwriter output_bufferprompt pdf_base64response llm_text_outscanned_pages_textj page_contente combined_textr'r'r(extract_text_from_pdf\s        $       o   z/EnhancedDocumentExtractor.extract_text_from_pdf || excel_filepath separatorc s zitj|ddd}d}|D]W\}|d|d7}dtfddjD}|d d tj|D}g} D]\} } |d d t| |D} | | qC|d |g|d 7}q|WSt y} zt d|dt| dt| WYd} ~ Sd} ~ ww)ai Converts an Excel file to a formatted text file, maintaining table structure. Args: excel_filepath (str): Path to the Excel file. separator (str, optional): Separator character between columns. Defaults to " || ". Returns: str: Formatted text representation of the Excel file Nr) sheet_nameheaderrVz --- Sheet: r8cs*g|]}tdtdd|jDqS)2css|] }tt|VqdSN)r+rp)r;xr'r'r( szOEnhancedDocumentExtractor.extract_text_from_excel...)minmaxvalues)r;coldfr'r(r=s*zEEnhancedDocumentExtractor.extract_text_from_excel..css"|] \}}|d|VqdS.css&|]\}}t|d|VqdSr)rp)r;cellrr'r'r(rs  r9zError extracting from Excel rUzError processing Excel: )pd read_excelitemsfillnaastyperpcolumnsjoinzipiterrowsraror[r\) r&rr all_sheetsrr col_widths header_line data_lines_row data_linerr'rr(extract_text_from_excels0       z1EnhancedDocumentExtractor.extract_text_from_excel csv_filepathc Cszzt|}|d}|t}|jddWSty<}ztd|dt|dt|WYd}~Sd}~ww)z Extract text from CSV file. Args: csv_filepath (str): Path to the CSV file Returns: str: Formatted text representation of the CSV file rVF)indexzError extracting from CSV rUzError processing CSV: N) rread_csvrrrp to_stringror[r\)r&rrrr'r'r(extract_text_from_csvs   z/EnhancedDocumentExtractor.extract_text_from_csv txt_filepathc Cs.zt|ddd }|WdWS1swYWdStyqz"t|ddd}|WdWYS1s@wYWYdStyp}ztd|dt|dt|WYd}~YSd}~wwty}ztd |dt|dt|WYd}~Sd}~ww) z Extract text from text file. Args: txt_filepath (str): Path to the text file Returns: str: Text content of the file rr>)encodingNzlatin-1zError reading text file rUzError processing text file: z Error extracting from text file )rWriUnicodeDecodeErrorror[r\rp)r&rrsrr'r'r(extract_text_from_txt(s& ( .z/EnhancedDocumentExtractor.extract_text_from_txt image_pathc Csza|jsWdSt|d}t|d}Wdn1s!wYtj|d }dddddd d }| |d}|jj j d d d ddddddd||ddgdgd}|j djWSty}ztd|dt|dt|WYd}~Sd}~ww)z Extract text from image file using Claude. Args: image_path (str): Path to the image file Returns: str: Extracted text content z6Error: No Claude API key provided for image processingr6r>Nr7z image/jpegz image/pngz image/tiffz image/bmprr?r@rAzYou are an expert at extracting text from images. Extract ALL text visible in the image, preserving the original formatting as much as possible.rBrzExtract all text content from this image. Preserve formatting, structure, and layout as much as possible. If this appears to be a document, maintain the document structure.rCrrFrGrJrLrOrz!Error extracting text from image rUzError processing image: )r#rWrFrhrirjr]r^splitextlowergetrTrkrNrror[r\rp) r&r image_file image_data image_ext mime_type_map mime_typer}rr'r'r(extract_text_from_image@sP   z1EnhancedDocumentExtractor.extract_text_from_image word_pathtemp_dirc Csz.tjtj|d}tj||d}t||||}tj|r,t||WSt yT}zt d|dt |dt |WYd}~Sd}~ww)a# Extract text from Word document by converting to PDF first. Args: word_path (str): Path to the Word document temp_dir (str): Directory to store temporary PDF file Returns: str: Extracted text content rrzError extracting from Word rUz Error processing Word document: N) r]r^rr_rrrexistsremoveror[r\rp)r&rr base_name pdf_filenamerrr'r'r(extract_text_from_word~s     z0EnhancedDocumentExtractor.extract_text_from_word file_pathcCs<tj|d}|jD] \}}||vr|SqdS)z Determine the file type based on extension. Args: file_path (str): Path to the file Returns: str: File type category r7 unsupported)r]r^rrr%r)r&rext file_type extensionsr'r'r( get_file_types  z'EnhancedDocumentExtractor.get_file_typeNc Cs||}|durtj|}zC|dkr||WS|dkr$||WS|dkr/|||WS|dkr9||WS|dkrC||WS|dkrM| |WSd|WSt yx}zt d |d t |d t |WYd}~Sd}~ww) a Extract text from a single file based on its type. Args: file_path (str): Path to the file temp_dir (str): Temporary directory for intermediate files Returns: str: Extracted text content NrrrrrrzUnsupported file type: zError extracting text from rUzError processing file: )rr]r^dirnamerrrrrrror[r\rp)r&rrrrr'r'r(extract_text_from_files,        z0EnhancedDocumentExtractor.extract_text_from_fileroot_dirc CsRg}t|D]\}}}|D]}tj||}||}|dkr%||qq|S)z Recursively find all supported document files in a directory tree. Args: root_dir (str): Root directory to search Returns: List[str]: List of file paths r)r]walkr^rrra) r&r found_filesrootdirsfilesrsrrr'r'r(find_documents_recursivelys   z4EnhancedDocumentExtractor.find_documents_recursively cache_keyc Cs|r||jvrtd||j|Stj|s#idd|dfS||}|s2idd|fStdt|di}g}g}t }|D]l}zHtdtj |d |j |tj |d } | r| d s| ||<||td tj |n||td tj |WqIty} ztd|d| ||WYd} ~ qId} ~ wwt |} |siddf} ntdd|D} |dd t|d| d| f} |r| |j|<| S)a Recursively extract text from all supported documents in a directory tree. Args: root_dir (str): Root directory containing documents to process cache_key (str, optional): Key for caching results Returns: Tuple[Dict[str, str], bool, str]: - Dictionary mapping file paths to extracted text - Success status - Status message z$Using cached extraction results for Fz Directory z does not existz%No supported document files found in r:z supported files to processz Processing z...)rErrorz!Successfully extracted text from zFailed to extract text from zError processing rUNz5No text was successfully extracted from any documentscss|]}t|VqdSr)r+)r;rr'r'r(rszMEnhancedDocumentExtractor.extract_documents_text_recursive..Tz documents (z total characters) in )r$r[r\r]r^rrr+rnowr_rr startswithrarosumr)r&rr all_filesdocuments_textsuccessful_files failed_files start_timerrrprocessing_timeresult total_charsr'r'r( extract_documents_text_recursivesJ         z:EnhancedDocumentExtractor.extract_documents_text_recursive)r3)rr)__name__ __module__ __qualname____doc__rpr)intr-boolr2rrrrrrrrrrrrrrr'r'r'r(r*s  *>#2rrr*cCst|S)z Factory function to create an EnhancedDocumentExtractor instance. Args: llm_api_key (str): API key for Claude/Gemini Returns: EnhancedDocumentExtractor: Initialized extractor instance )r)rr'r'r("create_enhanced_document_extractor(s r)+rr]rdsysjsonrFpathlibrtypingrrrrrmrXpandasrdocx2pdfrnltk nltk.tokenizergoogle.generativeai generativeairgoogle.generativeai.typesr typesr!loggingPILr r basicConfigINFO getLoggerrr[downloadrrrprr'r'r'r(sF