xinference.client.handlers.ChatModelHandle.generate

ChatModelHandle.generate(prompt: str, generate_config: PytorchGenerateConfig | None = None) → Completion | Iterator[CompletionChunk]

Creates a completion for the provided prompt and parameters via RESTful APIs.

Paramètres:

• prompt (str) – The user’s message or user’s input.

• generate_config (Optional["PytorchGenerateConfig"]) – Additional configuration for the chat generation. « PytorchGenerateConfig » -> Configuration for pytorch model

Renvoie:

Stream is a parameter in generate_config. When stream is set to True, the function will return Iterator[« CompletionChunk »]. When stream is set to False, the function will return « Completion ».

Type renvoyé:

Union[« Completion », Iterator[« CompletionChunk »]]

Lève:

RuntimeError – Fail to generate the completion from the server. Detailed information provided in error message.