-
Notifications
You must be signed in to change notification settings - Fork 215
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: one-pass IVF_PQ accelerated builds (#3001)
This feature improves disk IO dependence, but it is quite limited. This only works if the index type is IVF_PQ, and it will not work efficiently for local PQ in the future (unless we store _all_ the PQ models in VRAM). Importantly, this allows us to bypass local temp storage for storing residuals. However, this still stores PQ codes locally temporarily due to how we've implemented accelerator support, but these are much smaller (exact ratio depends on params). I tested on my local machine, which is sufficiently fast that the accelerated builds are mostly IO limited (but IO is also fast). I used wikipedia-40M New feature disabled: ![results_static_20241011_224535_plot_dataset_wikipedia-few-queries_k_10](https://github.com/user-attachments/assets/9a9285e1-1814-4215-a4c9-2a3f3a16c874) ivf training time: 52s ivf transform time: 89s pq training time: 18s pq assignment time: 143s create_index rust time: 8.9s New feature enabled: ![results_static_20241011_203303_plot_dataset_wikipedia-few-queries_k_10](https://github.com/user-attachments/assets/9d94f50b-e3b6-42f8-8357-3cb477e6279b) combined training time: 63.7s (not actually sure why this is faster, but it's not the big part anyway) combined transform time: 158.8s create_index rust time: 8.6s Improvement should be more noticeable for bigger datasets, as usual.
- Loading branch information
Showing
4 changed files
with
455 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.