Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve extractor retries #449

Open
sudo-suhas opened this issue Dec 1, 2022 · 0 comments
Open

Improve extractor retries #449

sudo-suhas opened this issue Dec 1, 2022 · 0 comments
Assignees

Comments

@sudo-suhas
Copy link
Contributor

Is your feature request related to a problem? Please describe.
If an API call fails with a temporary error while running an extractor, we do not retry it. It could be improved.

Describe the solution you'd like
We should provide optional retries for extractors similar to sinks where the extractor call will be retried if the error returned is an instance of plugins.RetryError.

Describe alternatives you've considered
Each extractor could manage its own retries. But this does not seem to be a clean approach and is also likely to get missed when a new extractor is being added.

Additional context
We could make the retrier implementation available to extractors so that they can retry a specific intermediate step with limited retries without restarting the entire extract process.

@sudo-suhas sudo-suhas changed the title Improve extractor retries feat: improve extractor retries Dec 1, 2022
@sudo-suhas sudo-suhas changed the title feat: improve extractor retries Improve extractor retries Dec 1, 2022
@ishanarya0 ishanarya0 self-assigned this May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants