-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
response_schema parameter leads to inconsistent and incorrect output #541
Comments
It seems like your response schema lacks multiple moves (e.g., move1, move2, etc.) and does not include orderType or currentOrder.
You can refer to the documentation link below for more clarity. |
Thanks for the response. However, note that there is no requirement that the response schema should support multiple moves or have all of the fields you've mentioned. At the end of the day a simple schema of just After testing with your schema, I see that adding the
|
I have run into a similar issue. Here's a minimal example import google.generativeai as genai
import typing_extensions as typing
class Card(typing.TypedDict):
word: str
category: str
definition: list[str]
example: list[str]
prompt = "Create a German vocabulary flashcard. Include fields 'word', 'category', 'definition' and 'example'"
model = genai.GenerativeModel(
"gemini-1.5-flash",
generation_config={
"candidate_count": 1,
"response_mime_type": "application/json",
"response_schema": Card, # <<<<<<<<<<<<<<< Comment this out
},
system_instruction=prompt,
)
response = model.generate_content("befassen")
print(response.text)
#> {"word": "befassen (sich mit etwas / jemanden) [sich beschäftigen] (verb) (reflexive verb) [to deal with, to be concerned with,
#> to occupy oneself with] (transitive verb) [to busy oneself with] (intransitive verb) [to have to do with] (informal) [to have a
#> go at] (transitive verb) [to touch] (intransitive verb) [to touch upon] (transitive verb) [to grip, to hold, to grasp] (intransitive
#> verb) [to apply oneself] (informal) [to get stuck into] (intransitive verb) [to get to grips with] (intransitive verb) [to take up]
#> (transitive verb) [to engage in] (transitive verb) [to involve oneself in] (transitive ver... After commenting out the "response_schema" line, the result looks much more reasonable:
This started happening about 10 days ago. |
Thanks @kinto-b , I've seen same behavior where not sending the I tried your example in AI studio, and the results seem more reasonable. So perhaps there's a bug in the Python API? |
Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs. |
issue is not resolved. |
Description of the bug:
The Barista Bot on AI studio is designed to output JSON in a format specified in the User prompt. From the prompt:
This works without JSON mode enabled, and even with JSON mode enabled and a blank
response_schema
specified. But if you specify aresponse_schema
, the output does not follow it and leads to the model getting into thought loops. Theresponse_schema
I'm giving it:Actual vs expected behavior:
Expected (which can be reliably generated when JSON mode is disabled):
Actual (you can get it to generate consistently inconsistent output every time you re-run, this is a particularly obvious example for illustration):
Any other information you'd like to share?
This issue doesn't seem specific to the python package, but I did reproduce it via API from my Python app. Sorry if I should have logged this in a better place!
The text was updated successfully, but these errors were encountered: