Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating/copying a collection fails randomly #66

Open
raffaeler opened this issue Sep 9, 2024 · 2 comments
Open

Creating/copying a collection fails randomly #66

raffaeler opened this issue Sep 9, 2024 · 2 comments

Comments

@raffaeler
Copy link

I am trying to use CreateCollectionAsync specifying the initFromCollection parameter to clone a collection.
I run the code under the debugger with all the .NET exceptions enabled.
In this situation, sometimes the CreateCollectionAsync succeed (no exceptions) but the new collection is not created. The great majority of the times, the collection is instead copied successfully.

In the output window I can see Copy ok (see the code below) to testify the CreateCollectionAsync has been called and it was successful.

    public async Task<bool> CopyCollectionAsync(
        string sourceCollection,
        string targetCollection,
        bool deleteTargetCollectionIfExists,
        bool createTargetCollectionIfNotExists,
        TimeSpan timeout)
    {
        var client = CreateClient();

        var targetExists = await client.CollectionExistsAsync(targetCollection);
        if (!createTargetCollectionIfNotExists && !targetExists)
        {
            return false;
        }

        if (deleteTargetCollectionIfExists && targetExists)
        {
            try
            {
                await client.DeleteCollectionAsync(targetCollection);
            }
            catch (Exception err)
            {
                Debug.WriteLine($"CopyCollectionAsync: error in deleting target collection: {err.ToString()}");
                return false;
            }
        }

        if (createTargetCollectionIfNotExists)
        {
            var sourceConfig = await GetCollectionConfig(sourceCollection);

            try
            {
                var spars = sourceConfig.Params;
                await client.CreateCollectionAsync(targetCollection,
                    spars.VectorsConfig.Params,
                    spars.ShardNumber,
                    spars.ReplicationFactor,
                    spars.WriteConsistencyFactor,
                    spars.OnDiskPayload,
                    sourceConfig.HnswConfig,
                    sourceConfig.OptimizerConfig,
                    sourceConfig.WalConfig,
                    sourceConfig.QuantizationConfig,
                    sourceCollection,                   // initial data (source)
                    spars.ShardingMethod,
                    spars.SparseVectorsConfig,
                    timeout,
                    default);

                Debug.WriteLine("Copy ok");
            }
            catch (Exception err)
            {
                Debug.WriteLine($"CopyCollectionAsync: error copying the collection: {err.ToString()}");
                return false;
            }
        }
        else
        {
            await foreach (var entry in GetAllAsync(sourceCollection))
            {
                if(!await Upsert(targetCollection,
                    entry.Value,
                    entry.Payload ?? new Dictionary<string, object?>()))
                {
                    return false;
                }
            }
        }

        return true;
    }
  • Qdrant container version: 1.11.3, build: 9fa86106
  • Qdrant client version: <PackageReference Include="Qdrant.Client" Version="1.11.0" />
  • .NET 8

Is this a bug?
Any clue / workarounds?

@russcam
Copy link
Collaborator

russcam commented Sep 12, 2024

Thanks for opening @raffaeler. If I understand your description correctly, the client call always succeeds, but sometimes the collection has not been created on the server?

@raffaeler
Copy link
Author

Yes @russcam, correct.
Succeeding with a failure behind the scenes is very alarming.
FYI In the output window of the debugger I don't see swallowed exceptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants