U h3@sdZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZddlmZe d eee efd Ze d eee e ee fd ZGd d d ZGdddeeefZGdddeeefZGdddeZdS)zSagemaker InvokeEndpoint API.N)abstractmethod) AnyDictGenericIteratorListMappingOptionalTypeVarUnion)CallbackManagerForLLMRun)LLM)pre_init)enforce_stop_tokens INPUT_TYPE)bound OUTPUT_TYPEc@s<eZdZdZeddddZddddZedd d ZdS) LineIteratoraParse the byte stream input. The output of the model will be in the following format: b'{"outputs": [" a"]} ' b'{"outputs": [" challenging"]} ' b'{"outputs": [" problem"]} ' ... While usually each PayloadPart event from the event stream will contain a byte array with a full json, this is not guaranteed and some of the json objects may be split acrossPayloadPart events. For example: {'PayloadPart': {'Bytes': b'{"outputs": '}} {'PayloadPart': {'Bytes': b'[" problem"]} '}} This class accounts for this by concatenating bytes written via the 'write' function and then exposing a method which will return lines (ending with a ' ' character) within the buffer via the 'scan_lines' function. It maintains the position of the last read position to ensure that previous bytes are not exposed again. For more details see: https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/ N)streamreturncCst||_t|_d|_dS)Nr)iter byte_iteratorioBytesIObufferread_pos)selfrrO/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/llms/sagemaker_endpoint.py__init__0s  zLineIterator.__init__rcCs|S)Nrrrrr__iter__5szLineIterator.__iter__cCs|j|j|j}|rJ|dtdkrJ|jt|7_|ddSzt|j}Wn,tk r|j|j j kr~YqYnXd|krq|jdt j |j |ddqdS)N Z PayloadPartrZBytes)rseekrreadlineordlennextr StopIteration getbuffernbytesrSEEK_ENDwrite)rlinechunkrrr__next__8s  zLineIterator.__next__)__name__ __module__ __qualname____doc__rrr"r1rrrrrsrc@s\eZdZUdZdZeeed<dZeeed<e e e e dddZ e e edd d Zd S) ContentHandlerBasezHandler class to transform input from LLM to a format that SageMaker endpoint expects. Similarly, the class handles transforming output from the SageMaker endpoint to a format that LLM class expects. z text/plain content_typeaccepts)prompt model_kwargsrcCsdS)zTransforms the input to a format that model can accept as the request Body. Should return bytes or seekable file like object in the format specified in the content_type request header. Nr)rr9r:rrrtransform_inputksz"ContentHandlerBase.transform_input)outputrcCsdS)z[Transforms the output from the model to string that the LLM class expects. Nr)rr<rrrtransform_outputssz#ContentHandlerBase.transform_outputN)r2r3r4r5r7r str__annotations__r8rrrbytesr;rr=rrrrr6Ls r6c@seZdZdZdS)LLMContentHandlerzContent handler for LLM class.N)r2r3r4r5rrrrrAzsrAc@seZdZUdZdZeed<dZeed<dZ eed<dZ e eed<e ed<d Z eed <dZe eed <dZe eed <Gd ddZeeedddZeeeefdddZeedddZdee eee eeedddZdS)SagemakerEndpointaSagemaker Inference Endpoint models. To use, you must supply the endpoint name from your deployed Sagemaker model & the region where it is deployed. To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html If a specific credential profile should be used, you must pass the name of the profile from the ~/.aws/credentials file that is to be used. Make sure the credentials / roles used have the required policies to access the Sagemaker endpoint. See: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html Nclient endpoint_name region_namecredentials_profile_namecontent_handlerF streamingr:endpoint_kwargsc@seZdZdZdS)zSagemakerEndpoint.ConfigZforbidN)r2r3r4extrarrrrConfigsrL)valuesrc Cs|ddk r|Szzddl}z@|ddk r<|j|dd}n|}|jd|dd|d<Wn,tk r}ztd |W5d}~XYnXWntk rtd YnX|S) z.Dont do anything if client provided externallyrCNrrG)Z profile_namezsagemaker-runtimerF)rFzCould not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid.zRCould not import boto3 python package. Please install it with `pip install boto3`.)getboto3SessionrC Exception ValueError ImportError)clsrMrOsessionerrrvalidate_environments2  z&SagemakerEndpoint.validate_environmentr cCs|jpi}d|jid|iS)zGet the identifying parameters.rEr:)r:rE)r _model_kwargsrrr_identifying_paramss z%SagemakerEndpoint._identifying_paramscCsdS)zReturn type of llm.Zsagemaker_endpointrr!rrr _llm_type(szSagemakerEndpoint._llm_type)r9stop run_managerkwargsrc Ksv|jpi}||}|jpi}|j||}|jj}|jj} |jr|rz||jjf|j ||jjd|} t | d} d} | D]@} t | } | dd}|dk rt||}| |7} ||qz| WStk r}ztd|W5d}~XYnXn~z"|jjf|j ||| d|}Wn2tk rH}ztd |W5d}~XYnX|j|d}|dk rnt||}|SdS) afCall out to Sagemaker inference endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. Example: .. code-block:: python response = se("Tell me a joke.") ) EndpointNameBody ContentTyper_rDoutputsrNz.Error raised by streaming inference endpoint: )r^r_r`Acceptz$Error raised by inference endpoint: )r:rJrHr;r7r8rIrCZ$invoke_endpoint_with_response_streamrErjsonloadsrNrZon_llm_new_tokenrQrRZinvoke_endpointr=)rr9r[r\r]rXZ_endpoint_kwargsbodyr7r8respiteratorZcurrent_completionr/Z resp_outputrVresponsetextrrr_call-sT       "    zSagemakerEndpoint._call)NN)r2r3r4r5rCrr?rEr>rFrGr rArIboolr:rrJrLrrWpropertyrrYrZrr rjrrrrrB~s2 A    $ rB)r5rrcabcrtypingrrrrrrr r r Zlangchain_core.callbacksr Z#langchain_core.language_models.llmsr Zlangchain_core.utilsrZlangchain_community.llms.utilsrr>rfloatrrr6rArBrrrrs ,    :.