o hhL;@s6ddlZddlZddlZddlZddlmZddlmZddlZddl Z ddl Z ddl m Z e e dZe ddZe dZe d d Ze d d Zdd lmZeeedZdZedadZeeZeZeZe jedZe je jdde e!Z"ddZ#ddZ$ddZ%ddZ&ddZ'dd Z(d!d"Z)dS)#N)bulk) Elasticsearch) load_dotenvANTHROPIC_API_KEYANTHROPIC_MODELzclaude-3-5-haiku-latestOPENAI_API_KEY OPENAI_MODELz Qwen/Qwen3-Next-80B-A3B-Instruct OPENAI_URLz#https://api.deepinfra.com/v1/openai)OpenAI)api_keybase_urlopen_llmzhttp://localhost:9200(Az1yGMw8rORRk87fO38h9fpmwqJ6ZY9IRA1ZGjEo)r z4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformatcCs:d}td|dt|tj||ddgd}|jjS)Nz embed-v4.0z*Inside encode_cohere function with model: z and search_documentfloat)textsmodel input_typeembedding_types)loggerinfolencoembed embeddingsr)chunksco_modelresponser 8/var/www/html/minaions-tender/ai-engine/chat_with_rfp.py encode_cohere5sr"c Cstd||}tjj|drdSt|}t|}td|tjj ||d|ddi}t j |rdt| }t|}Wdn1sPwYg} g} t |dkrb|D]d} i} t| d tr| d D]} | r| r| | | d | d <t| }| |qqn#| d r| d r| | d | d | d <t| }| |g|| d <|| d | d qb| r]td t | d t| }t |t | krtdt |dt | d|Sg}d}d}t| D]\}}zk|t |kr^||dur^||}t |dkrLi|d|i}d |vr>|d r>|d|t||d|d7}n-td|d|d7}ntd|dt ||d7}n td|d|d7}Wqty}ztd|d||d7}WYd}~qd}~wwtd|d|d|rVzHtt||dd d!d"d#\}}td$|d%|rtd&t |d%|dd'D] }td(|qtjj|dtd)|W|Sty@}zMtd*|td+|jdd'D] }td,|qztj|d}|d-}td.|d/Wn td0Ytjj|dWYd}~|Sd}~wtyU}z td1|d}~wwtd2|Std3|Std4||S)5Nz"Creating the Elasticsearch index: index zindex file for ES: settingsmappings)r$r&r'rquestiontagNameanswerz Processing z questions for embedding...zERROR: Vector count (z ) doesn't match question count ()i title_vectorr$)_op_type_index_id_sourcezSkipping document z: missing tagNamez: invalid vector dimension z: missing or null vectorzError processing document z: z Prepared z documents for indexing, z failed<iX)r$request_timeout max_retriesinitial_backoff max_backoffzSuccessfully indexed z documentszFailed to index z Failed item: zDone indexing for zBulk indexing error: zError details:z - countzDespite errors, z$ documents were successfully indexedz3Could not determine how many documents were indexedz'Unexpected error during bulk indexing: zNo valid documents to indexz#No valid questions found to processz$Data file doesn't exist. File name: )rrlower es_clientindicesexistsopenjsonloadcreateospathisfiler isinstanceliststripappendcopydeepcopyr" enumeratestr ExceptionrrefreshBulkIndexErrorerrorsr:)bid_idindex_config_file data_file bid_indexf index_config tag_responsesdata_json_filedocs questionsnew_docsfaqnew_faqqd title_vectorsrequestssuccessful_docs failed_docsidocvectorrequeste success_count failed_itemsitemerrorcount_response indexed_countr r r! es_index_dataAs                        rpcs`t|ddd }t|}Wdn1swYdd|Dfdd|ddDS) Nrzutf-8)encodingcSsi|] }|d|dqS)r)r*r ).0rlr r r! sz'get_matched_answers..cs,g|]}|ddvr|ddqS)r0r)r )rshittag_answer_mapr r! s z'get_matched_answers..hits)r?r@rA) es_response json_filepathrV json_datar rvr!get_matched_answerss   r}c Cs*td|tjj|drtd|dn td|d|d}dd iid d |id d i}tj|d|ddgidd}g}td|t|dddkrtdt||} tj || ddd} td| td| t | j D] \} } | | | j q~td||S)NzEntered ES Query for client: r#u✅ z index exists in ESu⚠️ z index does not exist in ESr script_score match_allz@cosineSimilarity(params.query_vector, doc['title_vector']) + 1.0 query_vector)sourceparams)queryscriptincludesr))sizerr0)r$bodyzData File path is ryu,✅ Positive responses returned in ES searchz rerank-v3.5)r documentstop_nrzAnswer doc list shows zReranked response list shows zFinal FAQ list is: )rrr<r=r>searchrr}rrerankrLresultsrIr$) rquery_embeddings client_idrRrTr script_queryrz final_contextanswer_doc_listreranked_responseidxresr r r! run_es_querysH    rcCsJtd|d|g}tjj|dr#t|g}t|||||}|S)z< Get context for a given query using Elasticsearch. z"Entered get_intent for client id: z, With user input: r#)rrr<r=r>r;r"r)textrRrrT context_listtext_embeddingsr r r! get_contexts  rcCstd||r |}nd|}d|d|d}tdkr9tjjtddd d |d gd }td |jdj StdkrZt j j jt dd d d |d gd}td |jdjjSdS)a_ Get response from Claude LLM using the context and user query. Args: query (str): The user's question context_list (list): List of context documents retrieved from Elasticsearch client_id (str): The client identifier Returns: str: Claude's response to the user query based on the context z&Getting Minaions response for client: z zYYou are an AI assistant helping with RFP (Request for Proposal) questions. CONTEXT: z USER QUERY: a Provide a helpful, accurate, and concise response to the user query based only on the information provided in the context above. If the context doesn't contain relevant information to answer the query, say "I don't have enough information to answer that question." DO NOT make up information not present in the context. Return ONLY the response to the USER QUERY with no additional commentary. claudeig?zYou are an AI assistant helping with RFP (Request for Proposal) questions. Respond only with information found in the provided context.user)rolecontent)r max_tokens temperaturesystemmessagesz(Minaions response generated successfullyrr r)rrN)rrjoin llm_model claude_clientrrB claude_modelrropenaichat completionsrchoicesmessage)rrr no_es_queryformatted_contextpromptrr r r!get_claude_response$s<     rcCs6|r t||||}|St||||}t||||}|S)aN Main function to handle the chat flow with RFP context retrieval and Claude response. Args: query (str): The user's question bid_id (str): The bid identifier for Elasticsearch index client_id (str): The client identifier Returns: str: Claude's response to the user query )rr)rrRrrTrmerged_txt_fileclaude_responserr r r! chat_with_rfp_s  r)*r@rCrJcohereelasticsearch.helpersr elasticsearchrredis anthropicloggingdotenvrgetenvrrrrr rr rr< CO_API_KEYClientrrCLAUDE_API_KEY Anthropicr basicConfigINFO getLogger__name__rr"rpr}rrrrr r r r!sL             5 ;