Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming doesn't work #1

Open
laszlovandenhoek opened this issue Sep 23, 2024 · 0 comments
Open

Streaming doesn't work #1

laszlovandenhoek opened this issue Sep 23, 2024 · 0 comments

Comments

@laszlovandenhoek
Copy link

Even Azure models that support streaming won't do it; the entire response is always returned in one chunk.

I have found no way to enable streaming using configuration, and from the code it doesn't seem possible. The problem appears to be with the can_stream property of the llm.Model class. Even if you define it using a config.yaml, it is ignored by the llm-azure plugin. AzureChat extends the OpenAI Chat, which in turn extends llm.Model. In Chat, can_stream is True by default, but this doesn't take effect because AzureChat doesn't call super().__init__(), so it becomes effectively False for all Azure models.

I propose to check config.yaml for a can_stream key, use it if present, and assume True otherwise. I will submit a PR shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant