Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase RDS connection pool max capacity in Prod #332

Merged
merged 1 commit into from
Jan 17, 2024
Merged

Increase RDS connection pool max capacity in Prod #332

merged 1 commit into from
Jan 17, 2024

Conversation

chuckwondo
Copy link
Collaborator

Also:

  • increase step function state concurrency, which the increased connection pool capacity should now handle
  • add missing retry configuration for steps in ingestion workflow to increase resilience
  • set duplicateHandling to "skip" for both WV03 collections to allow rerunning ingestion without reingesting existing granules
  • reduce background queue message limit to avoid reaching concurrency quota for Lambda executions
  • increase queue throttle to increase ingestion rate (in combination with previous point, this should still keep us within Lamba quota)
  • reduce default granule batch size for queuing to 1000 to ensure we won't encounter QueueGranules timeouts with "skip" duplicate handling (due to extra DB queries by DiscoverGranules to check for duplicates)

@chuckwondo
Increase RDS connection pool max capacity in Prod
bfedbf1 
Also:

- increase step function state concurrency, which the increased
  connection pool capacity should now handle
- add missing retry configuration for steps in ingestion workflow
  to increase resilience
- set duplicateHandling to "skip" for both WV03 collections to
  allow rerunning ingestion without reingesting existing granules
- reduce background queue message limit to avoid reaching concurrency
  quota for Lambda executions
- increase queue throttle to increase ingestion rate (in combination with
  previous point, this should still keep us within Lamba quota)
- reduce default granule batch size for queuing to 1000 to ensure we
  won't encounter QueueGranules timeouts with "skip" duplicate handling
  (due to extra DB queries by DiscoverGranules to check for duplicates)
krisstanton
krisstanton approved these changes Jan 16, 2024
View reviewed changes
Copy link
Collaborator

@krisstanton krisstanton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Good find on that DB connection pool setting!

@chuckwondo chuckwondo merged commit be0a82e into main Jan 17, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krisstanton krisstanton krisstanton approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chuckwondo @krisstanton