ADBC driver for DataFusion

[tokoko]

What feature or improvement would you like to see?

I started an adbc driver for datafusion in adbc-datafusion repo based on the rust dummy driver in this repo. It's mostly functional with support for get_objects, sql and substrait execution.

I'm not sure what's the most appropriate home for the driver though. Should it go here in arrow-adbc repo, the main datafusion repository or maybe as an independent repository as a datafusion-contrib?

[tokoko]

fyi @alamb, curious what you think

[lidavidm]

I think we would like projects to maintain their drivers (as DuckDB does), but the Rust target is also shifting. It's possible we should maintain it in -contrib or in tree until the Rust APIs have settled a bit before trying to upstream it so there is a reasonably stable target to follow.

[lidavidm]

@mbrobbel maybe something to explore is versioning the Rust API definitions separately as well so there is a very stable target instead of churning the version number every 2 months for no reason

[mbrobbel]

in tree until the Rust APIs have settled a bit

I agree with this. Keeping it in tree helps to understand how changes to core traits impact existing drivers, and drivers must be updated as part of PRs that change these traits.

maybe something to explore is versioning the Rust API definitions separately as well so there is a very stable target instead of churning the version number every 2 months for no reason

I agree it would be better to track this separately. There are a few things to consider:

• I guess this complicates the release process a little bit?
• As long as we're still using 0.x.y versions, applying non-breaking minor bumps is trivial for dependents, especially when they are using something like Dependabot.
• We are going to have to release whenever arrow-rs releases a new major version (we are using arrow-rs types in our public API). Dependents using both arrow-rs and adbc crates are blocked from updating their arrow related dependencies until we release with the latest arrow-rs major version.

[lidavidm]

Ah, I see (wrt arrow-rs types). In that case maybe there's not much value in separating the releases; we can just do minor version bumps.

[tokoko]

thanks, I'll open a PR here then.

[tokoko]

Only just now realized I was linking to a private repo in the description 🤦‍♂️ anyway.. PR is up at #2267