U h'@stddlmZmZmZmZmZmZmZmZddl m Z ddl m Z er`ddl mZmZddlmZGddde ZdS) ) TYPE_CHECKINGAnyDict GeneratorListMappingOptionalUnion)CallbackManagerForLLMRun)LLM)RESTfulChatModelHandleRESTfulGenerateModelHandle)LlamaCppGenerateConfigc seZdZUdZeed<eeed<eeed<eeefed<deeeeedfdd Z e ed d d Z e e eefd d dZ deeeeeeeedddZdedeeeedeeddfdddZZS) Xinferencea`Xinference` large-scale model inference service. To use, you should have the xinference library installed: .. code-block:: bash pip install "xinference[all]" If you're simply using the services provided by Xinference, you can utilize the xinference_client package: .. code-block:: bash pip install xinference_client Check out: https://github.com/xorbitsai/inference To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers Example: To start a local instance of Xinference, run .. code-block:: bash $ xinference You can also deploy Xinference in a distributed cluster. Here are the steps: Starting the supervisor: .. code-block:: bash $ xinference-supervisor Starting the worker: .. code-block:: bash $ xinference-worker Then, launch a model using command line interface (CLI). Example: .. code-block:: bash $ xinference launch -n orca -s 3 -q q4_0 It will return a model UID. Then, you can use Xinference with LangChain. Example: .. code-block:: python from langchain_community.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", model_uid = {model_uid} # replace model_uid with the model UID return from launching the model ) llm.invoke( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024, "stream": True}, ) To view all the supported builtin models, run: .. code-block:: bash $ xinference list --all client server_url model_uid model_kwargsNrrrc szddlm}WnRtk rbzddlm}Wn,tk r\}ztd|W5d}~XYnXYnX|pji}tjf|||d|jdkrtd|jdkrtd|||_ dS)Nr) RESTfulClientzCould not import RESTfulClient from xinference. Please install it with `pip install xinference` or `pip install xinference_client`.rzPlease provide server URLzPlease provide the model UID) xinference.clientr ImportErrorZxinference_clientsuper__init__r ValueErrorrr)selfrrrre __class__G/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/llms/xinference.pyr\s.  zXinference.__init__)returncCsdS)zReturn type of llm.Z xinferencerrrrr _llm_typeszXinference._llm_typecCsd|jid|jid|jiS)zGet the identifying parameters.rrrrr"rrr _identifying_paramsszXinference._identifying_params)promptstop run_managerkwargsr!c Ks|j|j}|di}|j|}|r0||d<|rf|drfd}|j||||dD] }||7}qT|S|j||d} | ddd Sd S) aqCall the xinference model and return the output. Args: prompt: The prompt to use for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Returns: The generated string by the model. generate_configr&stream)modelr%r'r)r%r)choicesrtextN)rZ get_modelrgetr_stream_generategenerate) rr%r&r'r(r,r)Zcombined_text_outputtoken completionrrr _calls"    zXinference._call)r r r)r,r%r'r)r!c csz|j||d}|D]b}t|tr|dg}|r|d}t|tr|dd} |d} |rn|j| |j| d| VqdS) a^ Args: prompt: The prompt to use for generation. model: The model used for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Yields: A string token. r-r.rr/r+Zlogprobs)r3verbose log_probsN)r2 isinstancedictr0Zon_llm_new_tokenr6) rr,r%r'r)Zstreaming_responsechunkr.choicer3r7rrr r1s&     zXinference._stream_generate)NN)NN)NN)__name__ __module__ __qualname____doc__r__annotations__rstrrrpropertyr#rr$rr r5r rr1 __classcell__rrrr r sD H  #   . rN)typingrrrrrrrr Zlangchain_core.callbacksr Z#langchain_core.language_models.llmsr rr r Zxinference.model.llm.corerrrrrr s (