o hh-'@s\dZddlZddlZddlZddlZddlZddlmZddlmZeddl Z ddl Z ddl Z ddl m Z mZddlZddlZddlmZddlZddlZddlmZe de d d Zed Zed d ZedZeddZeddZddl m!Z!e!eedZ eZ"dZ#e j$edZ%ddZ&d%ddZ'ddZ(dd Z)d!d"Z*d#d$Z+dS)&aC Document Processing Service This service processes documents (PDFs, TXTs) stored in S3: - Extracts text from documents - Merges text into a single file - Chunks text using LLMs for better semantic organization - Generates summaries using LLMs - Saves results back to S3 - Provides an API endpoint for processing requests N)datetime) load_dotenv) sent_tokenize word_tokenize)convert)extract_RFP_text_compatiblepunkt punkt_tab'AIzaSyCzr6L3E8yywy8Ls2errRBOPx740VcjV1gANTHROPIC_API_KEYANTHROPIC_MODELzclaude-3-5-haiku-latestOPENAI_API_KEY OPENAI_MODELz Qwen/Qwen3-Next-80B-A3B-Instruct OPENAI_URLz#https://api.deepinfra.com/v1/openai)OpenAI)api_keybase_urlopen_llm)rcCs tt|S)z Count words in text. Args: text (str): Text to count words in Returns: int: Number of words in text )lenr)textr6/var/www/html/minaions-tender/ai-engine/bid_queries.py count_wordsFs rdc Csg}t|}g}d}|D]:}t|}|||kr=|r"|d|t|dkr.|ddn|} | |g}td|}q ||||7}q |rQ|d||S)a3 Split text into overlapping chunks for LLM processing. Args: text (str): Text to split into chunks chunk_size (int): Target size of each chunk in words overlap (int): Number of words to overlap between chunks Returns: list: List of text chunks r N)rrappendjoinr) r chunk_sizeoverlapchunks sentences current_chunkcurrent_word_countsentencesentence_word_countoverlap_sentencesrrrcreate_primary_chunksRs"     r)c CszFdddddddd|dg}tdkr(tjjtd|d }|jd jd }ntd kr?tjj jt |d }|j d j jd }dd|DWSt yf}ztd|tjgWYd}~Sd}~ww)z Send chunk to LLM for processing into smaller, semantically coherent chunks. Args: chunk (str): Text chunk to process Returns: list: List of processed sub-chunks userzYou have to work as a Data Engineer who converts unstructured data into structured format for making it usable for ML Training.)rolecontent assistantz[Sure, I will help you in text data retructuring. Please let me know the exact requirements.aGreat! So you have to analyze a text segment and break it into smaller chunks following these rules: 1. Each chunk must be more than 400 words; 2. Each chunk must maintain complete semantic meaning; 3. Never break in the middle of a sentence or logical thought; 4. If the input contains any headers, titles or section names, headings or subheadings: - Identify such contextual content - Prepend these relevant headers or titles or section name to each chunk to maintain hierarchical context; 5. Ensure proper handling of: - Lists (keep items together when contextually related) - Tables (keep with their captions and context) - Quotes (preserve complete quotes in single chunks when possible) - Citations (keep with their referenced text); 6. Please delimt different chunks with this delimiter: '============break============'. Only create the chunks of the text and use the original text. DO NOT make up any text or content on your own. And please do not add any explanations in the output. Here is the text to process: claudei)model max_tokensmessagesrz============break============r)r/r1cSsg|] }|r|qSr)strip.0chunkrrr sz*process_chunk_with_llm..z!Error processing chunk with LLM: N) llm_model claude_clientr1creater,rsplitopenaichat completionsrchoicesmessage Exceptionprintsysstdoutflush)r5promptr? sub_chunkserrorrrrprocess_chunk_with_llm|s6  rHcCs:t||t\}}|rt|||fStd|||fS)a& Process all PDF Excel and Doc files from a directory and create a merged text file. Args: input_dir (str): Directory path for input files output_dir (str): Directory path for output files Returns: bool: True if successful, False otherwise zError: )rr rA) input_dir output_dirsuccessr?rrrextract_RFP_texts  rLc Cstj|d}zt|ddd }|}Wdn1swYWnty5td|YdSw|sCtdtj dStd tj t |}g}t |D]&\}}td |d d t |d tj t |}||td qVdd|D}ddt |D} tj|d} t| ddd} tj| | dddWdn1swYtd| tj dS)z Process the merged text file into chunks using LLM. Args: output_dir (str): Directory path within bucket for output files Returns: bool: True if successful, False otherwise z merged.txtrzutf-8)encodingNzError: File not found at path: Fz'Failed to read merged text file from S3zCreating primary chunks...z Processing chunk z of z...cSsg|]}|r|qSrrr3rrrr6sz%process_text_file..cSs*g|]\}}d|d|g|gdqS)tagrO)tagNamequestionanswer question_negr)r4ichunk_for_jsonrrrr6 s z chunks.jsonwr)indent ensure_asciiz'Successfully saved BID Chunks Json to: T)ospathropenreadFileNotFoundErrorrArBrCrDr) enumeraterrHextendtimesleepjsondump) rJ file_pathfilerprimary_chunksall_processed_chunksrUr5processed_chunks json_dataoutput_json_filefrrrprocess_text_filesF          rmcCsNtj|d}tj|ddt||\}}|r!t|}|rdSdSt|dS)z Process a document processing task from the queue. Args: task_data (dict): Task data including client_id, request_id, and configuration tender_analysisT)exist_okF)rZr[rmakedirsrLrmrA)bid_dirrJrKerr_msg chunk_successrrr process_tasksrt)rr),__doc__rZrcrBiorardotenvr anthropicPyPDF2nltk nltk.tokenizerrpandaspdpdfkitdocx2pdfrbase64pathlibdocument_extractorrdownloadGEMINI_API_KEYgetenvr r r rrr;r claude_modelr7 Anthropicr8rr)rHrLrmrtrrrrsN               *< R