v0.10.16
Bug fixes
- fix: use 1/ln(m) as the
mL
value for better performance by @BubbleCal in #2239 - fix: various fixes to ensure that the v2 writer is used by the ray sink if requested by @westonpace in #2248
- fix: bugs in list codec when dealing with a list of empty lists or multiple lists in a single page by @westonpace in #2222
- fix: creating IVF_HNSW index fails with cosine metric type by @BubbleCal in #2235
- fix: build & search causes panic with empty partition for IVF_HNSW by @BubbleCal in #2172
- fix: clippy on main by @chebbyChefNEQ in #2254
- fix: copy arrays when placing them in the v2 writer's accumulation queue by @westonpace in #2249
- fix: out of bound panics in take rows by @chebbyChefNEQ in #2259
- fix: allow for all-null pages in btree by @westonpace in #2245
New Features
- feat: add use_experimental_writer to write_fragments API by @westonpace in #2226
- feat: use v2 writer api in ray data sink by @eddyxu in #2231
- feat(java): add LanceOperation, commitAppend for batch write by @LuQQiu in #2207
- feat(python): expose FragmentMetadata attrs and compaction repr by @wjones127 in #2191
- feat: update arrow to 51, datafusion to 37 by @westonpace in #2240
- feat: improvement of Ray sink API by @eddyxu in #2237
- feat: forbid creating index if num_sub_vectors doesn't divide dim by @BubbleCal in #2234
- feat: add create_index to vector index extensions by @chebbyChefNEQ in #2250
- feat: support delta merge for IVF_HNSW_SQ by @BubbleCal in #2132
- feat(java): add version latestVersion api by @LuQQiu in #2238
- feat: raise GCS object size limit to 2.5TB by @wjones127 in #2261
Performance Improvements
- perf: concurrent building for HNSW graph by @BubbleCal in #2210
- perf: v2 fsl decode perf fix and some benchmarking utilities by @westonpace in #2214
- perf: avoid copying of creating memory dist calculator by @BubbleCal in #2219
- perf: keep less edges in non-base levels to improve greedy search performance by @BubbleCal in #2244
Other changes
- chore: remove --preview from ruff invocations by @westonpace in #2220
- chore: fix formatting in test_schema.py by @westonpace in #2227
- chore: bump to ruff 0.4.1 by @eddyxu in #2229
- chore: bump minimal python version to 39 by @eddyxu in #2230
- chore: use VectorStorage in HNSWBuilder by @chebbyChefNEQ in #2242
- test: test write methods against more varied layouts by @wjones127 in #2233
Full Changelog: v0.10.15...v0.10.16