Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mongodb): implement data filter for mongodb source #529

Merged
merged 6 commits into from
Sep 22, 2024

Conversation

IlyaFaer
Copy link
Contributor

Closes #521

@IlyaFaer IlyaFaer marked this pull request as ready for review July 15, 2024 08:47
@IlyaFaer IlyaFaer requested a review from rudolfix July 15, 2024 08:47
@IlyaFaer IlyaFaer closed this Aug 14, 2024
@IlyaFaer IlyaFaer force-pushed the 521/feat-implement-mongo-data-filter branch from 6122c6a to 84a4400 Compare August 14, 2024 07:42
Copy link
Collaborator

@Pipboyguy Pipboyguy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no exception or warning raised when $lte < $gte for example:

    movies = mongodb_collection(
        collection="movies",
        incremental=dlt.sources.incremental(
        "released",
        initial_value="1917-01-01T00:00:00Z"
    ),
        filter_={
        "released": {
            "$lte": "1800-06-01T00:00:00Z"
        }
    }
    )

Is this something we want to test or leave as is?

Otherwise LGTM

movies = mongodb_collection(
collection="movies",
incremental=dlt.sources.incremental("runtime", initial_value=20),
filter_={"runtime": {"$gte": 20}},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be worth adding tests for when $lts < $gte filter for the same field or thinking about whether we should handle this in some way?

@rudolfix
Copy link
Contributor

@Pipboyguy thanks for checking this out! I think we should let people to put any condition they want even if that makes sense. we do the same with sql alchemy AFAIK

@rudolfix rudolfix merged commit 69a1c5e into master Sep 22, 2024
23 of 25 checks passed
@rudolfix rudolfix deleted the 521/feat-implement-mongo-data-filter branch September 22, 2024 18:11
neuromantik33 pushed a commit to neuromantik33/verified-sources that referenced this pull request Oct 5, 2024
* feat(mongodb): implement data filter for mongodb source
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add filter to mongodb source
3 participants