Using Sparse Vectors

[twchase]

Does anybody have any examples of using sparse data during upsert and/or querying of Pinecone? I can't find any examples using Pinecone.Net. Any ideas on what tokenizer to use for .NET that works well? I started going down this path below but have no clue what is expected or how to get it into the Upsert properly.

var sentence = "New York City";

var tokenizer = new BertBaseTokenizer();

var encoded = tokenizer.Encode(512, sentence);

var bertInput = new BertInput()
{
    InputIds = encoded.Select(t => t.InputIds).ToArray(),
    AttentionMask = encoded.Select(t => t.AttentionMask).ToArray(),
    TypeIds = encoded.Select(t => t.TokenTypeIds).ToArray()
};

[neon-sunset]

Hi @twchase,

I haven't had a use for sparse-dense vectors yet but since this library matches the official API surface almost 1:1, you can pretty much follow the official example in https://www.pinecone.io/learn/hybrid-search-intro/ (although I sympathize with the fact that syntax and dynamic typing of Python does not help with understanding the exact behavior).

When .Upsert'ing a vector, its sparse values are represented by an optional SparseValues property with corresponding Indices and Values arrays.

If a tokenizer of your choice produces a pair of int and float sequences represented by IEnumerable/List/T[], then you can try passing them to a new vector with

var vec = new Vector
{
    Id = "first",
    Values = denseVector,
    // You may need to .ToArray() the source data because choosing a wrong primitive is a very common mistake
    SparseValues = new() { Indices = sparseIndices, Values = sparseData }
};
await pinecone.Upsert(new[] { vec });

Unfortunately I can't advise on any tokenizers for this use case in C#/.NET, there should not be anything specific or significantly different from Python libraries though.