aiextract and parametric types

[cpfiffer]

Not quite sure what's up with this example, but aiextract doesn't seem to know what to do here:

using PromptingTools

# Return types
abstract type AbstractThing end
struct Dolphin <: AbstractThing end
struct NotDolphin <: AbstractThing end

# Wrapper type
struct ThingHolder{C<:AbstractThing}
    dolphin_or_not::C
end

# Wrapper type that works
struct ThingHolder2
    dolphin_or_not::Union{Dolphin,NotDolphin}
end

# Call aiextract
tester = "Look it's probably a dolphin."
a = aiextract(tester; return_type=ThingHolder)
b = aiextract(tester; return_type=ThingHolder2)

In both cases, I get a Dict:

julia> a.content
Dict{Symbol, Any} with 1 entry:
  :dolphin_or_not => "dolphin"

julia> b.content
Dict{Symbol, Any} with 1 entry:
  :dolphin_or_not => "dolphin"

I think my expectation here would be that I'd get a ThingHolder, but the fallback seems to be to a Dict. Anyway thought I'd type this one up in case it's unexpected behavior.

Here's the warnings indicating that the JSON could not be read:

┌ Warning: There was an error parsing the response: ArgumentError("invalid JSON at byte position 19 while parsing type AbstractThing: ExpectedOpeningObjectChar\n{\"dolphin_or_not\":\"dolphin\"}\n"). Using the raw response instead.
└ @ PromptingTools ~/.julia/packages/PromptingTools/nuNH1/src/llm_openai.jl:649
[ Info: Tokens: 76 @ Cost: $0.0 in 0.8 seconds
PromptingTools.DataMessage(Dict with keys: dolphin_or_not)

┌ Warning: There was an error parsing the response: MethodError(JSON3.read, (StructTypes.SingletonType(), UInt8[0x7b, 0x22, 0x64, 0x6f, 0x6c, 0x70, 0x68, 0x69, 0x6e, 0x5f, 0x6f, 0x72, 0x5f, 0x6e, 0x6f, 0x74, 0x22, 0x3a, 0x22, 0x64, 0x6f, 0x6c, 0x70, 0x68, 0x69, 0x6e, 0x22, 0x7d], 19, 28, 0x22, NotDolphin), 0x0000000000007b0a). Using the raw response instead.
└ @ PromptingTools ~/.julia/packages/PromptingTools/nuNH1/src/llm_openai.jl:649
[ Info: Tokens: 78 @ Cost: $0.0 in 0.7 seconds
PromptingTools.DataMessage(Dict with keys: dolphin_or_not)

[svilupp]

I’ll have a look later but there are two challenges:

1. not sure my simple JSON schema generator inspects parametric types
2. even if it did, I leverage JSON3 reading into types via StructTypes and I think you would need to define the rule how to pick the subtypes of the abstract thing (eg, a subfield that tells you which one to use). See: https://juliadata.org/StructTypes.jl/stable/#StructTypes.AbstractType

It might be easier to enable passing a vector of types and pass both to the “tools”. At the moment, I fix it to one return type only but theoretically the model can choose its tool.

EDIT: After the second read, I'm fairly confident that you're hitting the 2) highlighted above. JSON3 has no idea what to unpack into, eg,
MWE

using JSON3

abstract type AbstractThing end
struct Dolphin <: AbstractThing end
struct NotDolphin <: AbstractThing end

struct ThingHolder{C<:AbstractThing}
    dolphin_or_not::C
end

obj = ThingHolder(Dolphin())
str = JSON3.write(obj)
# "\"ThingHolder{Dolphin}(Dolphin())\""
JSON3.read(str, ThingHolder)

The third issue is that the "name" (type) of the struct without any fields is hard to capture in JSON.
Struct is effectively a collection of key=>value pairs, so it's encoded in JSON as "object" (=Dict).

See what PromptingTools creates as the signature for the above object:

PT.function_call_signature(typeof(obj))
## Dict{String, Any} with 2 entries:
##   "name"       => "ThingHolderDolphin_extractor"
##   "parameters" => Dict{String, Any}("properties"=>Dict{String, Any}("dolphin_or_not"=>Dict{String, Any}("properties"=>Dict{String, Any}(), "type"=>"object")), "required"=>["dolphin_or_not"], "type"=>"object")

Properties are effectively an empty Dict, because the Struct is empty.

So either add the necessary annotating fields for your struct to enable the StructType dispatch with abstract types or simply use Enum/vector of symbols, or just use aiclassify as a router (more "token efficient" and faster!)