U hcP@sVddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZddlmZmZmZmZdd lmZmZd ZGd d d eZGd dde eZ!GdddZ"Gddde"Z#Gddde#Z$Gddde"Z%Gddde"Z&Gddde"Z'Gddde'Z(GdddeZ)Gdd d ee)Z*dS)!N)abstractmethod)Enum)AnyDictListMappingOptional)CallbackManagerForLLMRun)BaseLLM) Generation LLMResult) BaseModel SecretStrroot_validator validator)convert_to_secret_strget_from_dict_or_env2c@sFeZdZdZdefeeeeddddZd ee e e eddd Z dS) AzureMLEndpointClientz AzureML Managed Endpoint client.N) endpoint_urlendpoint_api_keydeployment_nametimeoutreturncCs,|r|std||_||_||_||_dS)zInitialize the class.zXA key/token and REST endpoint should be provided to invoke the endpointN) ValueErrorrrrr)selfrrrrrM/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/llms/azureml_endpoint.py__init__szAzureMLEndpointClient.__init__)body run_managerkwargsrcKs\dd|jd}|jdkr$|j|d<tj|j||}tjj||d|jd}| }|S)zcall.application/jsonzBearer )z Content-Type Authorizationrzazureml-model-deploymentr)r) rrurllibrequestRequestrurlopengetrread)rr r!r"headersreqresponseresultrrrcall&s    zAzureMLEndpointClient.call)N) __name__ __module__ __qualname____doc__DEFAULT_TIMEOUTstrintrbytesrr rr/rrrrrs" rc@seZdZdZdZdZdZdS)AzureMLEndpointApiTypezAzure ML endpoints API types. Use `dedicated` for models deployed in hosted infrastructure (also known as Online Endpoints in Azure Machine Learning), or `serverless` for models deployed as a service with a pay-as-you-go billing or PTU. dedicatedrealtime serverlessN)r0r1r2r3r9r:r;rrrrr8@sr8c@seZdZUdZdZeeed<dZeeed<dZ eed<e eeddd Z e e ed d d Zejfeeeed ddZeejfeeedddZdS)ContentFormatterBasezZTransform request and response of AzureML endpoint to match with required schema. r# content_typeacceptszError while formatting response payload for chat model of type `{api_type}`. Are you using the right formatter for the deployed model and endpoint type?format_error_msg)promptrcCs6dddddddd}|D]\}}|||}q|S) z*Escapes any special characters in `prompt`z\\z\"z\bz\fz\nz\rz\t)\"    )itemsreplace)r@Z escape_mapZescape_sequenceZescaped_sequencerrrescape_special_charactersys z.ContentFormatterBase.escape_special_charactersrcCstjgS)zSupported APIs for the given formatter. Azure ML supports deploying models using different hosting methods. Each method may have a different API structure.r8r9rrrrsupported_api_typessz(ContentFormatterBase.supported_api_typesr@ model_kwargsapi_typercCs tdS)zFormats the request body according to the input schema of the model. Returns bytes or seekable file like object in the format specified in the content_type request header. N)NotImplementedError)rr@rPrQrrrformat_request_payloads z+ContentFormatterBase.format_request_payloadoutputrQrcCsdS)zFormats the response body according to the output schema of the model. Returns the data type that is received from the response. Nr)rrUrQrrrformat_response_payloadsz,ContentFormatterBase.format_response_payloadN)r0r1r2r3r=rr5__annotations__r>r? staticmethodrJpropertyrr8rNr9rrrSrr7r rVrrrrr<Ls.     r<c@sLeZdZdZeeedddZee ee dddZ e ee dd d Z d S) GPT2ContentFormatterzContent handler for GPT2rKcCstjgSNrLrMrrrrNsz(GPT2ContentFormatter.supported_api_typesrOcCs2t|}tdd|dgi|d}t|S)N input_stringrBinputs parametersr<rJjsondumpsr5encoderr@rPrQrequest_payloadrrrrSs  z+GPT2ContentFormatter.format_request_payloadrTc Cs^zt|dd}Wn<tttfk rR}zt|jj|d|W5d}~XYnXt|dS)Nr0rQtext raloadsKeyError IndexError TypeErrorrr?formatr rrUrQchoiceerrrrVs &z,GPT2ContentFormatter.format_response_payloadNr0r1r2r3rYrr8rNr5rr7rSr rVrrrrrZs rZcs4eZdZUdZdZeed<ddfdd ZZS)OSSContentFormatterz`Deprecated: Kept for backwards compatibility Content handler for LLMs from the OSS catalog.Ncontent_formatterrKcsttddS)Nz`OSSContentFormatter` will be deprecated in the future. Please use `GPT2ContentFormatter` instead. superrwarningswarnrM __class__rrrs zOSSContentFormatter.__init__ r0r1r2r3rurrWr __classcell__rrrzrrts  rtc@sLeZdZdZeeedddZee ee dddZ e ee dd d Z d S) HFContentFormatterz6Content handler for LLMs from the HuggingFace catalog.rKcCstjgSr[rLrMrrrrNsz&HFContentFormatter.supported_api_typesrOcCs.t|td|dg|d}t|S)NrBr]r`rdrrrrSs  z)HFContentFormatter.format_request_payloadrTc Csbzt|ddd}Wn<tttfk rV}zt|jj|d|W5d}~XYnXt|dS)Nrrfgenerated_textrgrhrjrprrrrVs &z*HFContentFormatter.format_response_payloadNrsrrrrr~s r~c@sLeZdZdZeeedddZee ee dddZ e ee dd d Z d S) DollyContentFormatterz*Content handler for the Dolly-v2-12b modelrKcCstjgSr[rLrMrrrrNsz)DollyContentFormatter.supported_api_typesrOcCs2t|}tdd|dgi|d}t|S)Nr\rB) input_datar_r`rdrrrrSs z,DollyContentFormatter.format_request_payloadrTc CsZzt|d}Wn<tttfk rN}zt|jj|d|W5d}~XYnXt|dS)NrrgrhrjrprrrrVs &z-DollyContentFormatter.format_response_payloadNrsrrrrrs rc@sLeZdZdZeeedddZee ee dddZ e ee dd d Z d S) CustomOpenAIContentFormatterzAContent formatter for models that use the OpenAI like API scheme.rKcCs tjtjgSr[)r8r9r;rMrrrrNsz0CustomOpenAIContentFormatter.supported_api_typesrOcCsrt|}|tjtjfkr:tdd|dg|di}n.|tjkrXtd|i|}ntd|dt |S)z/Formats the request according to the chosen apirrB)r\r_r@ `api_type` # is not supported by this formatter) r<rJr8r9r:rarbr;rr5rcrdrrrrSs$    z3CustomOpenAIContentFormatter.format_request_payloadrTc Cs|tjtjfkrnzt|dd}Wn<tttfk rb}zt|j j |d|W5d}~XYnXt |dS|tj krz(t|dd}t |tstdWn<tttfk r}zt|j j |d|W5d}~XYnXt |dt|d |d d d Std |ddS)zFormats responserrfrgNrhchoicesziEndpoint response is not well formed for a chat model. Expected `dict` but `{type(choice)}` was received.ri finish_reasonlogprobs)rr)riZgeneration_inforr)r8r9r:rarkrlrmrnrr?ror r; isinstancedictstripr)rprrrrV-s4&   & z4CustomOpenAIContentFormatter.format_response_payloadNrsrrrrr s rcs4eZdZUdZdZeed<ddfdd ZZS)LlamaContentFormatterzNDeprecated: Kept for backwards compatibility Content formatter for Llama.NrurKcsttddS)Nz`LlamaContentFormatter` will be deprecated in the future. Please use `CustomOpenAIContentFormatter` instead. rvrMrzrrrVs zLlamaContentFormatter.__init__r|rrrzrrOs  rc@s eZdZUdZdZeed<ejZ eed<e dZ e ed<dZ eed<eZeed<dZeed <d Zeed <dZeed <dZeeed <eddeedddZed eeedddZedeedddZedeeedddZed ddeee dddZ!dS)AzureMLBaseEndpointz Azure ML Online Endpoint models.rrendpoint_api_typerrrN http_client max_retriesrurPT)pre)valuesrcCsdtt|dd|d<t|dd|d<t|ddd|d<t|dd tj|d<t|d d tt|d <|S) NrZAZUREML_ENDPOINT_API_KEYrZAZUREML_ENDPOINT_URLrZAZUREML_DEPLOYMENT_NAMErrZAZUREML_ENDPOINT_API_TYPErZAZUREML_TIMEOUT)rrr8r9r5r4)clsrrrrvalidate_environs6 z$AzureMLBaseEndpoint.validate_environ) field_valuerrcCs:|d}||jkr6tdt|d|jd|d|S)z>Validate that content formatter is supported by endpoint type.rzContent formatter fz8 is not supported by this endpoint. Supported types are z but endpoint is .)r)rNrtype)rrrrrrrvalidate_content_formatters   z.AzureMLBaseEndpoint.validate_content_formatter)rrcCs,|dr|dd}|dr(td|S)z'Validate that endpoint url is complete./Nzinference.ml.azure.comz`endpoint_url` should contain the full invocation URL including `/score` for `endpoint_api_type='dedicated'` or `/completions` or `/chat/completions` for `endpoint_api_type='serverless'`)endswithr)rrrrrvalidate_endpoint_urls   z)AzureMLBaseEndpoint.validate_endpoint_urlcCsZ|d}|tjks|tjkr0|ds0td|tjkrV|dsV|dsVtd|S)zBValidate that endpoint api type is compatible with the URL format.rz/scorezEndpoints of type `dedicated` should follow the format `https://..inference.ml.azure.com/score`. If your endpoint URL ends with `/completions` or`/chat/completions`, use `endpoint_api_type='serverless'` instead.z /completionsz/chat/completionszEndpoints of type `serverless` should follow the format `https://..inference.ml.azure.com/chat/completions` or `https://..inference.ml.azure.com/chat/completions`)r)r8r9r:rrr;)rrrrrrrvalidate_endpoint_api_types&  z.AzureMLBaseEndpoint.validate_endpoint_api_type)alwayscCs@|d}|d}|d}|dt}t||||}|S)z?Validate that api key and python package exists in environment.rrrr)r)r4rZget_secret_value)rrrrZ endpoint_keyrrrrrrvalidate_clients    z#AzureMLBaseEndpoint.validate_client)"r0r1r2r3rr5rWr8r9rrrrrr4rr6rrrrurPrrrrrrr<rrrrrrrrrr_s4          rc@sdeZdZdZeeeefdddZeedddZ d e ee e ee e ee dd d ZdS) AzureMLOnlineEndpointaAzure ML Online Endpoint models. Example: .. code-block:: python azure_llm = AzureMLOnlineEndpoint( endpoint_url="https://..inference.ml.azure.com/score", endpoint_api_type=AzureMLApiType.dedicated, endpoint_api_key="my-api-key", timeout=120, content_formatter=content_formatter, ) rKcCs|jpi}d|jid|iS)zGet the identifying parameters.rrP)rPr)r _model_kwargsrrr_identifying_paramss z)AzureMLOnlineEndpoint._identifying_paramscCsdS)zReturn type of llm.Zazureml_endpointrrMrrr _llm_typeszAzureMLOnlineEndpoint._llm_typeN)promptsstopr!r"rc Ksv|jpi}|||r ||d<g}|D]B}|j|||j}|jj||d} |j| |j} || gq(t |dS)anRun the LLM on the given prompts. Args: prompts: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. Example: .. code-block:: python response = azureml_model.invoke("Tell me a joke.") r)r r!) generations) rPupdaterurSrrr/rVappendr ) rrrr!r"rrr@reZresponse_payloadrrrr _generates*  zAzureMLOnlineEndpoint._generate)NN)r0r1r2r3rYrr5rrrrrr r rrrrrrs  r)+raurllib.requestr%rxabcrenumrtypingrrrrrZ langchain_core.callbacks.managerr Z#langchain_core.language_models.llmsr Zlangchain_core.outputsr r Zlangchain_core.pydantic_v1r rrrZlangchain_core.utilsrrr4objectrr5r8r<rZrtr~rrrrrrrrrs.    / `B