LanceDB Destination

[Pipboyguy]

Description

This PR adds support for using LanceDB as a destination.

• Updated documentation
• Unit tests

Related Issues

• closes LanceDB Destination #1370

Additional Context

With this change, dlt users can easily load their transformed data into LanceDB for efficient vector similarity search, full-text search, and SQL querying. LanceDB's ability to store raw data alongside vector embeddings makes it a powerful destination for AI applications.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	db1e81d
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/667d4d36be836c0008cdcb5a

[akelad]

Update: if I change the lancedb_adapter to from dlt.destinations.impl.lancedb.lancedb_adapter import lancedb_adapter the script runs. The docs say from dlt.destinations.adapters import lancedb_adapter, and that gave me the TypeError

[Pipboyguy]

@akelad All configuration fields are optional now. Please see associated tests as well. Mind checking whether it works for you?

Provided you aren't computing embeddings, you should be able to run as is after a dlt init

[akelad]

@Pipboyguy I'll try it out, thanks. Can you also take a look at my other comments? Both Anuun and I get the TypeError: 'module' object is not callable when using from dlt.destinations.adapters import lancedb_adapter - that should be fixed.

[akelad]

Can confirm it works without defining the api/embedding api key

[Pipboyguy]

@akelad

I actually can't get the lancedb_adapter to work properly so far. I did a dlt init rest_api lancedb, which doesn't produce the code to use the lancedb_adapter, it just treats lancedb as a normal db. Not sure how dlt init works and whether it's even feasible to adjust that.

Not all fields contain search-relevant information, and fields may require distinct embedding functions, especially for multi-modal data. Users must explicitly specify source fields for embeddings in their schema to ensure optimal performance and accuracy in vector search operations.

This is why the adapter has to be written by the user that understands what they are trying to achieve. If we want the adapter auto generated for the example we'd have to add this to the init command manually for each example as you pointed out.

I'm not sure how to do this though. @rudolfix @sh-rp ?

[Pipboyguy]

I understood you can only pass a resource if you also want to embed, and not a full source - is that correct?

Actually it seems that you can. Like you demonstrated above ( also ran it on my end), the adapter seems to work just fine with the pokemon source.

I speak under correction here though. I actually wasn't aware the an adapter also works on sources.

The general flow at least in my experience is to use the adapter with resources.

[Pipboyguy]

Update: if I change the lancedb_adapter to from dlt.destinations.impl.lancedb.lancedb_adapter import lancedb_adapter the script runs. The docs say from dlt.destinations.adapters import lancedb_adapter, and that gave me the TypeError

Thanks for this catch @akelad !

This should be fixed now. Kindly test again?

[akelad]

The fixed import works now! About the dlt init command - that's fine with me, maybe we should add some sort of note either in the generated code or as a warning that you have to add the lancedb_adapter for embeddings? Because out of the box it will just act as a normal DB. @sh-rp what do you think?

Actually it seems that you can. Like you demonstrated above ( also ran it on my end), the adapter seems to work just fine with the pokemon source.

Actually the example I showed there only runs on a resource: pokemon_source.resources['pokemon']. I don't think it works properly with a full source, at least when I tried it last week it did run, but there was some weird behaviour. I think we need to make it extra clear in the docs that this is intended to be used with resources and not on a source level - I can try and make those docs updates.

[Pipboyguy]

@akelad thank you so much for the detailed investigation.

Apologies, you are right the following does indeed work:

    lancedb_adapter(pokemon_source.resources['pokemon'], embed=["name"])
    load_info = pipeline.run(pokemon_source)

whereas the following doesn't raise a warning or exception, but nothing is embedded:

    lancedb_adapter(pokemon_source, embed=["name"])
    load_info = pipeline.run(pokemon_source)

I've updated the docs to make it clear that the adapter should only be used on resources, and that no fields will be embedded unless explicitly listed in the adapter.