U hx@sjdZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZzddlmZmZmZmZWn,ek rddlmZmZmZmZYnXddlZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%e&e'Z(Gd d d eZ)Gd ddeddZ*GdddeZ+GdddeddZ,GdddZ-ee+e,e.fZ/GdddeZ0ee0e.fZ1Gddde-Z2ddddZ3dZ4d d!d"d#Z5d$d$d%d&d'Z6Gd(d)d)Z7d*d)d+d,d-Z8d.d/d+d0d1Z9d.d2d+d3d4Z:d5d6d7d8d9Z;ee eej%eej$gee+e,ffe e ej%e ej$gee+e,fffZsz@DynamicRunEvaluator._coerce_evaluation_result..)r.rr/zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r5z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got )r:r,r5 ValueErrorrballr)rNrkr5rler$rpr%_coerce_evaluation_results,   z-DynamicRunEvaluator._coerce_evaluation_resultzUnion[dict, EvaluationResults]rH)rEr5rLcsLd|kr6|}fdd|dD|d<tf|Sjtt|ddS)NrEcsg|]}j|dqS))r5)ru)rnrrNr5r$r% szBDynamicRunEvaluator._coerce_evaluation_results..T)r5rl)copyrDrur rC)rNrEr5cpr$rwr%_coerce_evaluation_resultss   z.DynamicRunEvaluator._coerce_evaluation_resultszMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list])rkr5rLcCs.t|tr|js||_|St|}|||Srm)r:r,r5_format_evaluator_resultr{)rNrkr5r$r$r%_format_results  z"DynamicRunEvaluator._format_resultrLcCs t|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rXhasattrrNr$r$r%is_async'szDynamicRunEvaluator.is_asyncrrGrIcCst|ds6t}|r$tdn||||St}d|j i}t |ddrbt |j |d<|j ||||dd}|||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rWtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.r6 session_idN experimentrun_idmetadataZlangsmith_extra)rrPget_event_loop is_running RuntimeErrorrun_until_completerQuuiduuid4idrarrrWr})rNrJrK running_loopr5rrkr$r$r%rO0s"   z DynamicRunEvaluator.evaluate_runrJrKcsrt|dst||IdHSt}d|ji}t|ddrJt|j|d<|j ||||ddIdH}| ||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rXNr6rrrr) rsuperrQrrrrarrrXr})rNrJrKr5rrk __class__r$r%rQRs    z!DynamicRunEvaluator.aevaluate_runcCs |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )rOrMr$r$r%__call__mszDynamicRunEvaluator.__call__rcCsd|jdS))Represent the DynamicRunEvaluator object.zrbrr$r$r%__repr__~szDynamicRunEvaluator.__repr__)N)F)N)N)N)rr r!r"rhrur{r}propertyrrOrQrr __classcell__r$r$rr%rTs6 "rTrUrWcCst|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rTrr$r$r% run_evaluators ri'r)objcCs,t|}t|tkr(|dtdd}|S)Nz...))reprlen_MAXSIZE)rsr$r$r% _maxsize_reprs rrC)inputsrLcCs&t|d}t|d}||dS)NrJrKr)rget)rZ run_truncatedZexample_truncatedr$r$r%r`sr`c@seZdZdZd"dddddZedd d d Zd#d d ddddZd$d d ddddZd%d d ddddZ dd ddZ e d ddddZ ddd ddd d!Z dS)&DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.NfCallable[[Sequence[Run], Optional[Example]], Union[_COMPARISON_OUTPUT, Awaitable[_COMPARISON_OUTPUT]]]zUOptional[Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]rVcCst|}|rt|}t||ddlm}|dk rR|j|td|_t|dd|_t |r|dk rlt d|j|td|_t|dd|_n8|jt t ttttgtf|td|_t|dd|_dS)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rrYNr[rrTr\)$_normalize_comparison_evaluator_funcrr^rZr_r`rXrarbrcrdrer rr rr r_COMPARISON_OUTPUTrWrgr$r$r%rhsB    z&DynamicComparisonRunEvaluator.__init__rjr~cCs t|dSrrrr$r$r%rsz&DynamicComparisonRunEvaluator.is_async Sequence[Run]rGrRrunsrKrLcCslt|ds6t}|r$tdn||||St}| |}|j ||||dd}| |||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rWrrtagsr) rrPrrrr acompare_runsrr _get_tagsrW_format_results)rNrrKrr5rrkr$r$r% compare_runss"   z*DynamicComparisonRunEvaluator.compare_runscsRt|ds|||St}||}|j||||ddIdH}||||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rXrrN)rrrrrrXr)rNrrKr5rrkr$r$r%rs    z+DynamicComparisonRunEvaluator.acompare_runscCs |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rNrrKr$r$r%rsz&DynamicComparisonRunEvaluator.__call__rcCsd|jdS)rzJszADynamicComparisonRunEvaluator._format_results..)rSr-r5r-zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r5zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got ) r:rRr5listziprbrCrrr)rNrkr5rmsgrtr$r$r%r>s2     z-DynamicComparisonRunEvaluator._format_results)N)N)N)N)rr r!r"rhrrrrrr staticmethodrrr$r$r$r%rs8   rr)rWrLcCst|S)z.Create a comaprison evaluator from a function.)rrr$r$r%comparison_evaluatorcsrrz|Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]csdt}dd|jDrHtfddDs^tdkr^dd}t|ntfd dDrd d gkrStrd d ddfdd }tdrt dn|j |_ |Sd dddfdd }tdrt dn|j |_ |SdS)NrJrKroutputsreference_outputscSs&g|]\}}|j|j|jfkr|qSr$kindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrnpnamepr$r$r%rxusz-_normalize_evaluator_func..c3s|]}|kVqdSrmr$rnrsupported_argsr$r%rq{sz,_normalize_evaluator_func..kInvalid evaluator function. Must have at least one positional argument. Supported positional arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3s|]}|kVqdSrmr$rrr$r%rqsrJrKrrGrfrIcsN|||r|jni|jpi|r&|jp(inidfddD}|IdHS)Nrc3s|]}|VqdSrmr$rnargZarg_mapr$r%rqsz>_normalize_evaluator_func..awrapper..rrrJrKargsrWpositional_argsrr%awrappers z+_normalize_evaluator_func..awrapperrrcsH|||r|jni|jpi|r&|jp(inidfddD}|S)Nrc3s|]}|VqdSrmr$rrr$r%rqsz=_normalize_evaluator_func..wrapper..rrrrr%wrappers z*_normalize_evaluator_func..wrapper rc signature parametersitemsrsrrrrdrrarrWsigrrrr$rWrrr%r]ms@          r]zUnion[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]csdt}dd|jDrHtfddDs^tdkr^dd}t|ntfd dDrd d gkrStrd d ddfdd }tdrt dn|j |_ |Sd dddfdd }tdrt dn|j |_ |SdS)NrrKrrrcSs&g|]\}}|j|j|jfkr|qSr$rrr$r$r%rxsz8_normalize_comparison_evaluator_func..c3s|]}|kVqdSrmr$rrr$r%rqsz7_normalize_comparison_evaluator_func..rrrc3s|]}|kVqdSrmr$rrr$r%rqsrrKrrGrrcsR|||r|jnidd|D|r*|jp,inidfddD}|IdHS)NcSsg|]}|jpiqSr$rrnrJr$r$r%rxszJ_normalize_comparison_evaluator_func..awrapper..rc3s|]}|VqdSrmr$rrr$r%rqszI_normalize_comparison_evaluator_func..awrapper..rrrKrrrr%rs  z6_normalize_comparison_evaluator_func..awrapperrrcsL|||r|jnidd|D|r*|jp,inidfddD}|S)NcSsg|]}|jpiqSr$rrr$r$r%rxszI_normalize_comparison_evaluator_func..wrapper..rc3s|]}|VqdSrmr$rrr$r%rqszH_normalize_comparison_evaluator_func..wrapper..rrrrr%rs  z5_normalize_comparison_evaluator_func..wrapperrrr$rr%rs@          rz;Union[EvaluationResults, dict, str, int, bool, float, list]zUnion[EvaluationResults, dict])rkrLcCst|tttfrd|i}nx|s.td|ndt|trdtdd|DsZtd|dd|i}n.t|trxd|i}nt|trntd ||S) Nr.zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got css|]}t|tVqdSrm)r:rC)rnxr$r$r%rq sz+_format_evaluator_result..z8Expected a list of dicts or EvaluationResults. Received .rErzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) r:rjr<r;rrrrsrrCrpr$r$r%r|s(       r|SUMMARY_EVALUATOR_Tcsdt}dd|jDrHtfddDsrtdkrrdd}rh|d d7}t|n^tfd dDrd d gkrSd dddfdd }tdrtdn|j |_ |SdS)NrexamplesrrrcSs&g|]\}}|j|j|jfkr|qSr$rrr$r$r%rx,sz0_normalize_summary_evaluator..c3s|]}|kVqdSrmr$rrr$r%rq2sz/_normalize_summary_evaluator..rrrz Received positional arguments c3s|]}|kVqdSrmr$rrr$r%rq>srrzSequence[schemas.Run]zSequence[schemas.Example]rH)rrrLcs^||dd|Ddd|Ddd|DdfddD}|}t|trV|St|S)NcSsg|] }|jqSr$)rrnrKr$r$r%rxJszA_normalize_summary_evaluator..wrapper..cSsg|]}|jpiqSr$rrr$r$r%rxKscSsg|]}|jpiqSr$rrr$r$r%rxLsrc3s|]}|VqdSrmr$rrr$r%rqNsz@_normalize_summary_evaluator..wrapper..)r:r,r|)rrrrkrrr%rDs    z-_normalize_summary_evaluator..wrapperr) rcrrrrsrrrrrar)rWrrrr$rr%_normalize_summary_evaluator)s2      r)>r" __future__rrPrcrabcrtypingrrrrrr r r r r Ztyping_extensionsrr^rZ pydantic.v1rrrr ImportErrorZpydanticlogging functoolsrZlangsmith.schemasrrrr getLoggerrr=rr&r,rDrFrCrfrRrrTrrrr`rrr]rr|rrr$r$r$r%sd  0     /  W E JJ