-
Notifications
You must be signed in to change notification settings - Fork 751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arrays and primitive types are not supported in structured output (native or otherwise) #5521
Comments
Structured outputs in OpenAI require an `object` schema object. By detecting this situation and wrapping always in a `Payload<T>(T Data)` record, we significantly improve the developer experience by making the API transparent to that limitation (which might even be a temporary one?). The approach works for both OpenAI as well as Azure Inference without native structured outputs. In order to signal a wrapped result to the `ChatCompletion<T>`, we use the `AdditionalProperties` dictionary with a non-standard `$wrapped` property which is a typical convention for JSON properties that are not intended for end-user consumption (like $schema). Fixes dotnet#5521
We could potentially do that, but it has other risks:
Based on this we've taken the decision (so far at least - it could change) to limit how much magic this library does around structured output. This means:
As such it's up to the developer to augment this behavior further if they want. For scalar outputs, they should provide their own wrapping object. And if the non-native prompt isn't sufficient, they should add their own further prompting. It's a tricky area and we'll only know if the design is right once people start using this at scale in hundreds of different app scenarios. If we go on to collect evidence from further feedback that a particular change is warranted, we'll certainly be open to making that change. Hope that seems reasonable! |
Hm. This is one of the areas that starts to feel like old EF where the LINQ you got was actually a VERY leaky abstraction that didn't hold in a ton of scenarios, requiring knowledge of what worked and what didn't. Here you see a T, where a large number of instances just won't work. Lots of frustration and unnecessary wrappers by everyone. I view it as a VERY low-hanging fruit, and something that I'm almost sure future OpenAI LLMs may likely solve natively too, since it just looks like an oversight (or a bug). I don't see how automatically wrapping (AND passing that as the schema you tell the LLM to use, either natively or via your existing prompt) can have vastly different results as it already can have with the existing mechanism. My PR shows how you basically pass the same schema (wrapped top-level) in both cases, so the delta in behavior would be the same as it could be today. Sure, this can end up in a package with a single extension method |
If you need an array (or list) of typed values from completion rather than a single object, things fail whether you're using native json schema support or not. These two tests fail, for example:
Except for the first test, the others don't even compile due to a constraint on the T being a
class
, but there is really no need for that to be the case and it seriously restricts the usefulness of the typed API, forcing the creation of intermediate types just to hold those values (i.e. a superfluousPayload<T>
).The native JSON schema in OpenAI does require the schema type to be an object (not an array or anything else). But this could be solved by having the API itself providing the wrapping object automatically, so that users fall more easily in the proverbial pit of success.
The text was updated successfully, but these errors were encountered: