o h$@sdZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z ddl m Z ddl mZddlmZejejded eejgd eeZGd d d Zd ededefddZdededefddZdS)z Main Document Processor This is the main application that orchestrates the entire document processing pipeline: 1. Recursive document discovery and text extraction 2. Document analysis using Claude LLM 3. Data formatting into required output formats N)DictAnyOptional)datetime)"create_enhanced_document_extractor)create_document_analyzer)create_data_formatterz4%(asctime)s - %(name)s - %(levelname)s - %(message)szdocument_processor.log)levelformathandlersc @sxeZdZdZdefddZ ddededed eeeffd d Z d eded eeeffd dZ d eee ffddZ dS)DocumentProcessorzH Main document processor that orchestrates the entire pipeline. claude_api_keycCs&||_t||_t||_t|_dS)z Initialize the document processor. Args: claude_api_key (str): API key for Claude N)r r extractorranalyzerr formatter)selfr rB/var/www/html/minaions-tender/ai-engine/main_document_processor.py__init__'s   zDocumentProcessor.__init__T input_dir output_dir use_cachereturnc Cst}tdtd|td|tj|dd|||id}ztdt}|r8|nd}|jj||d \}} } t|} | | | t |d |d d <| sot d | d|d<| |d<|WStdt |dtj |d} t| ddd} tj|| dddWdn1swYtdt}|j|}t|}|j|}d| t ||d|d d<tdt |dtd|ddtd t}|j||}t|}d| |d!|d d"<t|}d|d<t|d#<| |d$<||d%<td&|td'||WSty`}z"t d(|d|d<t||d<t|d#<|WYd}~Sd}~ww))a Process all documents in the input directory and generate outputs. Args: input_dir (str): Directory containing documents to process output_dir (str): Directory to save output files use_cache (bool): Whether to use caching for extraction Returns: Dict[str, Any]: Processing results and statistics z%Starting document processing pipelinezInput directory: zOutput directory: T)exist_ok) start_timerrstepsz)Step 1: Extracting text from documents...N) cache_key)successmessagedurationdocuments_foundr extractionzText extraction failed: Frerrorz!Successfully extracted text from z documentszextracted_text.jsonwzutf-8)encoding)indent ensure_asciiz*Step 2: Analyzing documents with Claude...)rrdocuments_analyzedsummaryanalysiszSuccessfully analyzed zAverage confidence: avg_confidence.2fz(Step 3: Formatting and saving results...)rr output_files formattingend_timetotal_durationr-z.Document processing completed successfully in zOutput files saved to: z"Error during document processing: )rnowloggerinfoosmakedirs isoformatr extract_documents_text_recursive total_secondslenr"pathjoinopenjsondumpranalyze_documents_batchget_analysis_summaryrsave_formatted_data Exceptionstr)rrrrrprocessing_results step1_startrdocuments_textextraction_successextraction_messagestep1_durationextracted_text_pathf step2_startanalysis_resultsstep2_durationanalysis_summary step3_startr-step3_durationr0errrprocess_documents3s             z#DocumentProcessor.process_documents file_pathc Cstd|tj|sdd|dSz*|j|}|dr(d|dWS|j ||}||i}|j ||}d||dWSt ya}zt d|dt|dWYd }~Sd }~ww) a  Process a single document file. Args: file_path (str): Path to the document file output_dir (str): Directory to save output files Returns: Dict[str, Any]: Processing results zProcessing single document: FzFile not found: )rr"ErrorT)ranalysis_resultr-z"Error processing single document: N)r2r3r4r:existsrextract_text_from_file startswithranalyze_documentrrArBr"rC)rrTrtextrVrMr-rRrrrprocess_single_documents&    z)DocumentProcessor.process_single_documentcCs|jjS)z Get the list of supported file types. Returns: Dict[str, list]: Dictionary of supported file extensions by category )rsupported_extensions)rrrrget_supported_file_typessz*DocumentProcessor.get_supported_file_typesN)T) __name__ __module__ __qualname____doc__rCrboolrrrSr\listr^rrrrr "s   n(r rrapi_keycCsPt|}|||}|drtdtd|d|Std|d|S)z Example function showing how to process a directory of documents. Args: input_dir (str): Directory containing documents output_dir (str): Directory to save outputs api_key (str): Claude API key rz"Processing completed successfully!zOutput files: r-Processing failed: r")r rSprint)rrre processorresultsrrrprocess_directorys rjrTcCslt|}|||}|dr+tdtd|ddtd|dddd Std |d d S) z Example function showing how to process a single document. Args: file_path (str): Path to document file output_dir (str): Directory to save outputs api_key (str): Claude API key rz Document processed successfully!zDocument type: rVdoc_typez Confidence: confidencer,rfr"N)r r\rg)rTrrerhrirrrprocess_single_files rm)rbr4sysr=loggingargparsetypingrrrrenhanced_document_extractorrdocument_analyzerrdata_formatterr basicConfigINFO FileHandler StreamHandlerstdout getLoggerr_r2r rCrjrmrrrrs.       3