Performance optimization of IVF-flat / select_k

[mfoerste4]

This PR is a followup to #2169. To enable IVF-flat with k>256 we need an additional select_k invocation which was unexpectedly slow. There are two reasons for that:

First problem is the data handed to select_k: The valid data length per row is much smaller than the conservative maximum that could be achieved by probing the N largest probes. Therefore each query row contains roughly ~50% dummy values. This is also the case for IVF-PQ, but did not show up as prominent due to the second reason.

The second problem, and also a difference to the IVF-PQ algorithm - is that a 64bit payload data type is used for selectK. The performance of selectK with 64bit index type is significantly slower than with 32bit, especially when many elements are in the same range:

Benchmark                                                           Time             CPU   Iterations
-----------------------------------------------------------------------------------------------------
SelectK/float/uint32_t/kRadix11bitsExtraPass/1/manual_time       1.68 ms         1.74 ms          413 1357#200000#512
SelectK/float/uint32_t/kRadix11bitsExtraPass/3/manual_time       2.31 ms         2.37 ms          302 1357#200000#512#same-leading-bits
SelectK/float/int64_t/kRadix11bitsExtraPass/1/manual_time        5.92 ms         5.98 ms          116 1357#200000#512
SelectK/float/int64_t/kRadix11bitsExtraPass/3/manual_time        83.7 ms         83.8 ms            8 1357#200000#512#same-leading-bits
-----------------------------------------------------------------------------------------------------

The data distribution within a IVF-flat benchmark resulted in a select_k time of ~24ms.

scope:

• additional parameter added to select_k to optionally pass individual row lengths for every batch entry. This parameter is utilized by both IVF-Flat and IVF-PQ and results in a ~2x speedup (50 nodes out of 5000) of the final select_k.
• refactor ivf-flat search to work with 32bit indices by storing positions instead of actual indices. This allows to utilize 32bit index type select_k for ~10x speedup in the final select_k.

FYI @tfeher @achirkin

not in scope:

• General optimization of select_k: In the current implementation there is no difference in the type of the payload and the actual index type. Especially the type of the histogram has a large effect on performance (due to the atomics).

[tfeher]

Thanks Malte for the PR! I am excited to see this is an important perf improvement for IVF-Flat with large-k! I just have two small comments.

[mfoerste4]

Thanks @tfeher for your review. I have pushed a bugfix and some minor cosmetic changes.

[tfeher]

Thanks Malte for the update, LGTM.

[mfoerste4]

@tfeher thanks again for the review. I have deactivated the new check for the cagra filter test and added #2234 to follow up.

[tfeher]

/merge