v0.14.2-beta.1

What's Changed

New Features 🎉

• feat: add fast search to to_table method by @chebbyChefNEQ in #2591

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592

Full Changelog: v0.14.1...v0.14.2-beta.1

v0.14.1

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in #2526
• feat(python): expose fast search flag in python by @eddyxu in #2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
• feat: add scalar bitmap index to lance by @raunaks13 in #2560
• feat: add a standalone method for calculating the PQ model by @westonpace in #2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575
• feat: support to set ef for search from python by @BubbleCal in #2569
• feat: dataset update accepts binary value by @eddyxu in #2579
• feat: do flat search if too many rows are filtered out by @BubbleCal in #2583
• feat: add label list index by @westonpace in #2581
• feat: allow passing in precomputed centroids to lance.util.KMeans by @jiachengdb in #2586
• feat: add standalone partition assignment operation by @westonpace in #2556

Bug Fixes 🐛

• fix: add support for list fields with names other than 'item' by @westonpace in #2580
• fix: propagate storage_options to v2 writer by @jiachengdb in #2578
• fix: be compatible with v1 index format by @BubbleCal in #2582
• fix: search panic with prefilter and HNSW by @BubbleCal in #2584

Performance Improvements 🚀

• perf: change scalar index to return RowIdTreeMap instead of u64 array by @westonpace in #2587

Full Changelog: v0.14.0...v0.14.1

v0.14.1-beta.2

What's Changed

New Features 🎉

• feat: support to set ef for search from python by @BubbleCal in #2569

Full Changelog: v0.14.1-beta.1...v0.14.1-beta.2

v0.14.1-beta.1

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in #2526
• feat(python): expose fast search flag in python by @eddyxu in #2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
• feat: add scalar bitmap index to lance by @raunaks13 in #2560
• feat: add a standalone method for calculating the PQ model by @westonpace in #2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575

Full Changelog: v0.14.0...v0.14.1-beta.1

v0.14.0

What's Changed

Breaking Changes 🛠

• fix: return correct sub index statistics by @BubbleCal in #2532
• feat: change to snapshot isolation by @wjones127 in #2537

New Features 🎉

• feat: enhance binary array encoding, make it the default by @westonpace in #2521
• feat: stable row id support in queries by @wjones127 in #2452
• feat: make it possible to create v2 fragments using the fragment API by @westonpace in #2534
• feat: add dictionary encoding to lance by @raunaks13 in #2409
• feat: provide a fast ANN search plan by @eddyxu in #2554
• feat: create standalone IVF training API in python by @westonpace in #2553
• feat: support u8 for existing distance types by @BubbleCal in #2501
• feat: reserve cores for IO by @chebbyChefNEQ in #2496
• feat: allow users to create a file writer without a schema by @westonpace in #2558

Bug Fixes 🐛

• fix: pytorch dot inference pass unwanted parameters by @eddyxu in #2539
• fix: show underlying cause of DynamoDB service error by @wjones127 in #2541
• fix: build HNSW with wrong distance type by @BubbleCal in #2533
• fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in #2547
• fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in #2564
• fix: avoid multiple threads loading same index partition by @albertlockett in #2559
• fix: failed to return index stats if the first partition is empty by @BubbleCal in #2565
• fix: propagate storage_options to create_index for reading precomputed partition file by @jiachengdb in #2568

Documentation 📚

• docs: artefact_management_doc by @tanaymeh in #2504

Performance Improvements 🚀

• perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in #2527

Other Changes

• refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path by @BubbleCal in #2469
• refactor: flat search to use datafusion top k by @eddyxu in #2535
• refactor: remove generic PQ impl and trait by @BubbleCal in #2529

New Contributors

• @jiachengdb made their first contribution in #2547
• @walterddr made their first contribution in #2382

Full Changelog: v0.13.0...v0.14.0

v0.14.0-beta.2

What's Changed

New Features 🎉

• feat: add dictionary encoding to lance by @raunaks13 in #2409
• feat: provide a fast ANN search plan by @eddyxu in #2554
• feat: create standalone IVF training API in python by @westonpace in #2553
• feat: support u8 for existing distance types by @BubbleCal in #2501
• feat: reserve cores for IO by @chebbyChefNEQ in #2496
• feat: allow users to create a file writer without a schema by @westonpace in #2558

Bug Fixes 🐛

• fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in #2564
• fix: avoid multiple threads loading same index partition by @albertlockett in #2559

Other Changes

• refactor: remove generic PQ impl and trait by @BubbleCal in #2529

New Contributors

• @walterddr made their first contribution in #2382

Full Changelog: v0.14.0-beta.1...v0.14.0-beta.2

v0.14.0-beta.1

What's Changed

Breaking Changes 🛠

• fix: return correct sub index statistics by @BubbleCal in #2532
• feat: change to snapshot isolation by @wjones127 in #2537

New Features 🎉

• feat: stable row id support in queries by @wjones127 in #2452
• feat: make it possible to create v2 fragments using the fragment API by @westonpace in #2534

Bug Fixes 🐛

• fix: pytorch dot inference pass unwanted parameters by @eddyxu in #2539
• fix: show underlying cause of DynamoDB service error by @wjones127 in #2541
• fix: build HNSW with wrong distance type by @BubbleCal in #2533
• fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in #2547

Performance Improvements 🚀

• perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in #2527

Other Changes

• refactor: flat search to use datafusion top k by @eddyxu in #2535

New Contributors

• @jiachengdb made their first contribution in #2547

Full Changelog: v0.13.1-beta.1...v0.14.0-beta.1

v0.13.1-beta.1

What's Changed

New Features 🎉

• feat: enhance binary array encoding, make it the default by @westonpace in #2521

Documentation 📚

• docs: artefact_management_doc by @tanaymeh in #2504

Other Changes

• refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path by @BubbleCal in #2469

Full Changelog: v0.13.0...v0.13.1-beta.1

v0.13.0

What's Changed

Breaking Changes 🛠

• fix: dot distance so kmeans can converge by @eddyxu in #2509

New Features 🎉

• feat: convert binary logical encoding/decoding to physical array encoding/page decoding by @raunaks13 in #2426
• feat: support stable row ids in Dataset::take_rows() by @wjones127 in #2447
• feat: enable aarch64-pc-windows builds by @NickDarvey in #2512
• feat: expose DatasetPreFilter, PreFilter, and FilterLoader to the public API by @westonpace in #2522

Bug Fixes 🐛

• fix: low recall if many partitions with only 1 row by @BubbleCal in #2487
• fix: allow row id only in v2 by @westonpace in #2481

Documentation 📚

• docs: flickr8k_dataset_creation_example by @tanaymeh in #2489
• docs(python): note multiprocessing incompatibility by @wjones127 in #2506

Performance Improvements 🚀

• perf: don't write precomputed partitions to trace file by @westonpace in #2493
• perf: use faster kmean find partition routing for pq assignment by @eddyxu in #2515

Other Changes

• refactor: combined capacity updates and decoding for all physical decoders by @raunaks13 in #2508

New Contributors

• @NickDarvey made their first contribution in #2512

Full Changelog: v0.12.3...v0.13.0

v0.13.0-beta.2

What's Changed

New Features 🎉

• feat: enable aarch64-pc-windows builds by @NickDarvey in #2512
• feat: expose DatasetPreFilter, PreFilter, and FilterLoader to the public API by @westonpace in #2522

Bug Fixes 🐛

• fix: allow row id only in v2 by @westonpace in #2481

Other Changes

• refactor: combined capacity updates and decoding for all physical decoders by @raunaks13 in #2508

New Contributors

• @NickDarvey made their first contribution in #2512

Full Changelog: v0.13.0-beta.1...v0.13.0-beta.2