v0.10.7: major bugfix for drop_columns, storage_options in Python

Bug fixes

❗ There was an bug with drop_columns(). If you've called this on your dataset, you should check if your dataset was affected by running dataset.validate(). If this raises an error, you can call dataset.delete("false") to force a repair operation on your dataset. Afterward it will work as expected.

• fix: remove data files with all dropped columns by @wjones127 in #2130

New Features

🚀 You can now configure object storage connection in the kwargs of lance.dataset() and lance.write_dataset() with storage_options. For example:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "access_key_id": "my-access-key",
        "secret_access_key": "my-secret-key",
        "session_token": "my-session-token",
    }
)

Read more in https://lancedb.github.io/lance/read_and_write.html#object-store-configuration

• feat(python): expose storage options by @wjones127 in #2131
• feat: extend datagen to cover more types by @westonpace in #2138
• feat: add a protobuf file describing encodings by @westonpace in #2137
• feat: add a basic encodings crate by @westonpace in #2139
• feat: support IVF_HNSW_SQ by @BubbleCal in #2136

Other Changes

• chore: expose dynamic projection on fragment API by @chebbyChefNEQ in #2144

Full Changelog: v0.10.6...v0.10.7