U h[$@sddlmZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZGdd d eeZGd d d eZGd d d e ZdS))Enum)AnyIteratorListOptional)CallbackManagerForLLMRun)LLM)GenerationChunk) BaseModel)enforce_stop_tokensc@seZdZdZdZdZdS)Devicez,The device to use for inference, cuda or cpucudacpuN)__name__ __module__ __qualname____doc__r rrrJ/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/llms/titan_takeoff.pyr sr c@sjeZdZUdZGdddZeed<ejZ eed<dZ eed<dZ e e ed <d Ze ed <d Ze ed <dS) ReaderConfigzAConfiguration for the reader to be deployed in Titan Takeoff API.c@seZdZdZdS)zReaderConfig.ConfigrN)rrrZprotected_namespacesrrrrConfigsrZ model_namedeviceZprimaryconsumer_groupNtensor_parallelimax_seq_lengthmax_batch_size)rrrrrstr__annotations__r r rrrrintrrrrrrrs    rcseZdZUdZdZeed<dZeed<dZ eed<dZ e ed <d Z e ed <ddddgfeeee eed fd d ZeedddZdeeeeeee edddZdeeeeeee eedddZZS) TitanTakeoffaTitan Takeoff API LLMs. Titan Takeoff is a wrapper to interface with Takeoff Inference API for generative text to text language models. You can use this wrapper to send requests to a generative language model and to deploy readers with Takeoff. Examples: This is an example how to deploy a generative language model and send requests. .. code-block:: python # Import the TitanTakeoff class from community package import time from langchain_community.llms import TitanTakeoff # Specify the embedding reader you'd like to deploy reader_1 = { "model_name": "TheBloke/Llama-2-7b-Chat-AWQ", "device": "cuda", "tensor_parallel": 1, "consumer_group": "llama" } # For every reader you pass into models arg Takeoff will spin # up a reader according to the specs you provide. If you don't # specify the arg no models are spun up and it assumes you have # already done this separately. llm = TitanTakeoff(models=[reader_1]) # Wait for the reader to be deployed, time needed depends on the # model size and your internet speed time.sleep(60) # Returns the query, ie a List[float], sent to `llama` consumer group # where we just spun up the Llama 7B model print(embed.invoke( "Where can I see football?", consumer_group="llama" )) # You can also send generation parameters to the model, any of the # following can be passed in as kwargs: # https://docs.titanml.co/docs/next/apis/Takeoff%20inference_REST_API/generate#request # for instance: print(embed.invoke( "Where can I see football?", consumer_group="llama", max_new_tokens=100 )) zhttp://localhostbase_urli porti mgmt_portF streamingNclient)r!r"r#r$modelscsrtj||||dzddlm}Wntk r@tdYnX||j|j|jd|_|D]}|j |q\dS)aInitialize the Titan Takeoff language wrapper. Args: base_url (str, optional): The base URL where the Takeoff Inference Server is listening. Defaults to `http://localhost`. port (int, optional): What port is Takeoff Inference API listening on. Defaults to 3000. mgmt_port (int, optional): What port is Takeoff Management API listening on. Defaults to 3001. streaming (bool, optional): Whether you want to by default use the generate_stream endpoint over generate to stream responses. Defaults to False. In reality, this is not significantly different as the streamed response is buffered and returned similar to the non-streamed response, but the run manager is applied per token generated. models (List[ReaderConfig], optional): Any readers you'd like to spin up on. Defaults to []. Raises: ImportError: If you haven't installed takeoff-client, you will get an ImportError. To remedy run `pip install 'takeoff-client==0.4.0'` )r!r"r#r$r) TakeoffClientzjtakeoff-client is required for TitanTakeoff. Please install it with `pip install 'takeoff-client>=0.4.0'`.)r"r#N) super__init__Ztakeoff_clientr' ImportErrorr!r"r#r%Z create_reader)selfr!r"r#r$r&r'model __class__rrr)ns& zTitanTakeoff.__init__)returncCsdS)zReturn type of llm.Z titan_takeoffr)r+rrr _llm_typeszTitanTakeoff._llm_type)promptstop run_managerkwargsr/c Ks\|jr.d}|j|||dD]}||j7}q|S|jj|f|}|d}|dk rXt||}|S)aCall out to Titan Takeoff (Pro) generate endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. run_manager: Optional callback manager to use when streaming. Returns: The string generated by the model. Example: .. code-block:: python model = TitanTakeoff() prompt = "What is the capital of the United Kingdom?" # Use of model(prompt), ie `__call__` was deprecated in LangChain 0.1.7, # use model.invoke(prompt) instead. response = model.invoke(prompt) )r1r2r3textN)r$_streamr6r%generater ) r+r1r2r3r4Z text_outputchunkresponser6rrr_calls   zTitanTakeoff._callc ks|jj|f|}d}|D]|}||j7}d|kr|drN) r%Zgenerate_streamdata startswithlensplitrstripr Zon_llm_new_tokenr6replace) r+r1r2r3r4r:bufferr6content_r9rrrr7s*     zTitanTakeoff._stream)NN)NN)rrrrr!rrr"rr#r$boolr%rrrr)propertyr0rrr;rr r7 __classcell__rrr-rr ,sL 2     .  1 r N)enumrtypingrrrrZlangchain_core.callbacksrZ#langchain_core.language_models.llmsrZlangchain_core.outputsr Zlangchain_core.pydantic_v1r Zlangchain_community.llms.utilsr rr rr rrrrs