U h9@stddlZddlZddlmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZGddde ZdS) N)AnyDictListOptionalcast) AgentAction AgentFinish)BaseCallbackHandler) LLMResult)parsecseZdZUdZdZeed<edZeed<dZeed<dZ eed <d3ee ee ee ed d fd d Z e ee feee d dddZee d dddZee d dddZee d dddZe ee fe ee fe d dddZe ee fe d dddZee d dd d!Ze ee fee d d"d#d$Zee e d%d&d'Zd4e e ee ee d d(d)d*Zee d dd+d,Zee d d-d.d/Zee d d0d1d2ZZ S)5ArgillaCallbackHandleraCallback Handler that logs into Argilla. Args: dataset_name: name of the `FeedbackDataset` in Argilla. Note that it must exist in advance. If you need help on how to create a `FeedbackDataset` in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html. workspace_name: name of the workspace in Argilla where the specified `FeedbackDataset` lives in. Defaults to `None`, which means that the default workspace will be used. api_url: URL of the Argilla Server that we want to use, and where the `FeedbackDataset` lives in. Defaults to `None`, which means that either `ARGILLA_API_URL` environment variable or the default will be used. api_key: API Key to connect to the Argilla Server. Defaults to `None`, which means that either `ARGILLA_API_KEY` environment variable or the default will be used. Raises: ImportError: if the `argilla` package is not installed. ConnectionError: if the connection to Argilla fails. FileNotFoundError: if the `FeedbackDataset` retrieval from Argilla fails. Examples: >>> from langchain_community.llms import OpenAI >>> from langchain_community.callbacks import ArgillaCallbackHandler >>> argilla_callback = ArgillaCallbackHandler( ... dataset_name="my-dataset", ... workspace_name="my-workspace", ... api_url="http://localhost:6900", ... api_key="argilla.apikey", ... ) >>> llm = OpenAI( ... temperature=0, ... callbacks=[argilla_callback], ... verbose=True, ... openai_api_key="API_KEY_HERE", ... ) >>> llm.generate([ ... "What is the best NLP-annotation tool out there? (no bias at all)", ... ]) "Argilla, no doubt about it." z%https://github.com/argilla-io/argillaREPO_URLz/issues ISSUES_URLzphttps://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.htmlBLOG_URLzhttp://localhost:6900DEFAULT_API_URLN) dataset_nameworkspace_nameapi_urlapi_keyreturnc sntzddl}|j|_Wntk r:tdYnXt|jtdkr`td|jd|dkrtddkrt d|j d |j }|dkrtd dkrt|jtd krd nd |_ t d|j d|j }z|j ||dWn>tk r,}ztd|d|jd|W5d}~XYnX||_|p@||_zXi}t|jtdkr|t d|jdtddi}|jjf|j|jd||_WnVtk r}z6td|d|jd|jd|jd|jd |W5d}~XYnXdd g}|d!d"|jjDkrPtd#|jd|jd$|d%d&d"|jjDd'|jd( i|_t d)|jddS)*a)Initializes the `ArgillaCallbackHandler`. Args: dataset_name: name of the `FeedbackDataset` in Argilla. Note that it must exist in advance. If you need help on how to create a `FeedbackDataset` in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html. workspace_name: name of the workspace in Argilla where the specified `FeedbackDataset` lives in. Defaults to `None`, which means that the default workspace will be used. api_url: URL of the Argilla Server that we want to use, and where the `FeedbackDataset` lives in. Defaults to `None`, which means that either `ARGILLA_API_URL` environment variable or the default will be used. api_key: API Key to connect to the Argilla Server. Defaults to `None`, which means that either `ARGILLA_API_KEY` environment variable or the default will be used. Raises: ImportError: if the `argilla` package is not installed. ConnectionError: if the connection to Argilla fails. FileNotFoundError: if the `FeedbackDataset` retrieval from Argilla fails. rNzTo use the Argilla callback manager you need to have the `argilla` Python package installed. Please install it with `pip install argilla`z1.8.0z#The installed `argilla` version is z} but `ArgillaCallbackHandler` requires at least version 1.8.0. Please upgrade `argilla` with `pip install --upgrade argilla`.ZARGILLA_API_URLz[Since `api_url` is None, and the env var `ARGILLA_API_URL` is not set, it will default to `z6`, which is the default API URL in Argilla Quickstart.ZARGILLA_API_KEYz1.11.0z admin.apikeyz owner.apikeyz[Since `api_key` is None, and the env var `ARGILLA_API_KEY` is not set, it will default to `z6`, which is the default API key in Argilla Quickstart.)rrz.Could not connect to Argilla with exception: 'z'. Please check your `api_key` and `api_url`, and make sure that the Argilla server is up and running. If the problem persists please report it to z as an `integration` issue.1.14.0zYou have Argilla z., but Argilla 1.14.0 or higher is recommended.Z with_recordsF)nameZ workspacez@`FeedbackDataset` retrieval from Argilla failed with exception `z+`. Please check that the dataset with name=z in the workspace=zz exists in advance. If you need help on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit z.. If the problem persists please report it to promptresponsecSsg|] }|jqSr.0fieldrrR/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/callbacks/argilla_callback.py sz3ArgillaCallbackHandler.__init__..z`FeedbackDataset` with name=z] had fields that are not supported yet for the`langchain` integration. Supported fields are: z/, and the current `FeedbackDataset` fields are cSsg|] }|jqSrrrrrrr szl. For more information on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit .zThe `ArgillaCallbackHandler` is currently in beta and is subject to change based on updates to `langchain`. Please report any issues to )super__init__Zargilla __version__ARGILLA_VERSION ImportErrorr osgetenvwarningswarnrZDEFAULT_API_KEYinit ExceptionConnectionErrorrrZ get_workspacer UserWarningZFeedbackDatasetZ from_argilladatasetFileNotFoundErrorrfields ValueErrorprompts) selfrrrrZrge extra_argsZsupported_fields __class__rrr#=s        * 6  zArgillaCallbackHandler.__init__) serializedr3kwargsrcKs$|jt|dp|d|idS)z.Save the prompts in memory when an LLM starts. parent_run_idrun_idN)r3updatestr)r4r9r3r:rrr on_llm_startsz#ArgillaCallbackHandler.on_llm_start)tokenr:rcKsdS)z)Do nothing when a new token is generated.Nr)r4r@r:rrron_llm_new_tokensz'ArgillaCallbackHandler.on_llm_new_token)rr:rc s|dr dS|jt|d}t||jD]$\}|jjfdd|Ddq*|jt|dt|jtdkr|j dS)z(Log records to Argilla when an LLM ends.r;Nr<cs g|]}d|jdiqS)r1rr)textstrip)rZ generationrrrr s z5ArgillaCallbackHandler.on_llm_end..recordsr) r3r>zip generationsr/ add_recordspopr r%push_to_argilla)r4rr:r3rIrrEr on_llm_ends  z!ArgillaCallbackHandler.on_llm_end)errorr:rcKsdS)z%Do nothing when LLM outputs an error.Nrr4rNr:rrr on_llm_errorsz#ArgillaCallbackHandler.on_llm_error)r9inputsr:rcKsHd|krD|jt|dp|dt|dtr6|dn|dgidS)aIf the key `input` is in `inputs`, then save it in `self.prompts` using either the `parent_run_id` or the `run_id` as the key. This is done so that we don't log the same input prompt twice, once when the LLM starts and once when the chain starts. inputr;r<N)r3r=r> isinstancelist)r4r9rQr:rrron_chain_starts z%ArgillaCallbackHandler.on_chain_start)outputsr:rc s.tfddt|dt|dfDs.dSjt|dp\ttjt|dg}|D]V\}}t|trj j ddt ||Ddqfj j d d || d igdqft|djkrjt|dt|djkr jt|dtjtd kr*j dS) zIf either the `parent_run_id` or the `run_id` is in `self.prompts`, then log the outputs to Argilla, and pop the run from `self.prompts`. The behavior differs if the output is a list or not. c3s|]}|jkVqdS)N)r3)rkeyr4rr sz6ArgillaCallbackHandler.on_chain_end..r;r<NcSs&g|]\}}d||ddiqS)r1rCrB)rD)rroutputrrrr s  z7ArgillaCallbackHandler.on_chain_end..rFr1 rBr)anyr>r3getrritemsrSrTr/rJrHjoinrDrKr r%rL)r4rVr:r3Zchain_output_keyZchain_output_valrrXr on_chain_ends:    z#ArgillaCallbackHandler.on_chain_endcKsdS)z+Do nothing when LLM chain outputs an error.NrrOrrron_chain_error8sz%ArgillaCallbackHandler.on_chain_error)r9 input_strr:rcKsdS)zDo nothing when tool starts.Nr)r4r9rbr:rrr on_tool_start<sz$ArgillaCallbackHandler.on_tool_start)actionr:rcKsdS)z.Do nothing when agent takes a specific action.Nr)r4rdr:rrron_agent_actionEsz&ArgillaCallbackHandler.on_agent_action)rZobservation_prefix llm_prefixr:rcKsdS)zDo nothing when tool ends.Nr)r4rZrfrgr:rrr on_tool_endIsz"ArgillaCallbackHandler.on_tool_endcKsdS)z&Do nothing when tool outputs an error.NrrOrrr on_tool_errorSsz$ArgillaCallbackHandler.on_tool_error)rCr:rcKsdSz Do nothingNr)r4rCr:rrron_textWszArgillaCallbackHandler.on_text)finishr:rcKsdSrjr)r4rlr:rrron_agent_finish[sz&ArgillaCallbackHandler.on_agent_finish)NNN)NN)!__name__ __module__ __qualname____doc__r r>__annotations__rrrrr#rrrr?rAr rM BaseExceptionrPrUr`rarcrrerhrirkrrm __classcell__rrr7rr sb +       2   r )r'r)typingrrrrrZlangchain_core.agentsrrZlangchain_core.callbacksr Zlangchain_core.outputsr Zpackaging.versionr r rrrrs