Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cli] add explicit paginators to rest_api template #1904

Closed
wants to merge 2 commits into from

Conversation

AstrakhantsevaAA
Copy link
Contributor

users have some issues with dlt init rest_api duckdb command, I set paginators explicitly, because they are not detected automatically for some reason. also reduced the amount of data for github api (request data for a week, not for month) because of rate limits

Copy link

netlify bot commented Oct 1, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 559f344
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66fbeecc09c7a20008b93b35

@burnash
Copy link
Collaborator

burnash commented Oct 1, 2024

I set paginators explicitly, because they are not detected automatically for some reason

Can you reproduce it? If yes, let's create an issue for this and fix it.
(but we still can explicitly define the paginators of course)

@burnash
Copy link
Collaborator

burnash commented Oct 1, 2024

If we want to explicitly define pagination, we'd need to update the tutoral as it says that we didn't explicitly specify pagination: https://dlthub.com/docs/tutorial/rest-api#pagination

@AstrakhantsevaAA
Copy link
Contributor Author

@burnash

Can you reproduce it? If yes, let's create an issue for this and fix it.

Yes, I can, unfortunately, for both sources :( Let's create an issue then.

Screenshot 2024-10-01 at 15 48 57

@burnash
Copy link
Collaborator

burnash commented Oct 2, 2024

Yes, I can, unfortunately, for both sources :( Let's create an issue then.

@AstrakhantsevaAA I think it's not an actual error, but a not very good warning message. Most likely we need to change the message. This Fallback pagiantor warning comes from the child (dependent) resource and informs that there's no paginator on this page.

@AstrakhantsevaAA
Copy link
Contributor Author

@burnash yes, I see it is just a warning, but also it is the cause of the rate limit error. Not sure why it is happening, but if i add pagination explicitly then the data loads without any problem and rate limit errors. It seems like it keeps requesting one page until the rate limit error, so you can see the list of these warnings (with pagination it takes 1-2 requests).

created the issue: #1915

@AstrakhantsevaAA AstrakhantsevaAA deleted the fix_init_restapi_template branch October 3, 2024 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants