Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load EcoPlaftform data from S3 to PostgreSQL #5

Open
aduverger opened this issue Oct 1, 2024 · 0 comments
Open

Load EcoPlaftform data from S3 to PostgreSQL #5

aduverger opened this issue Oct 1, 2024 · 0 comments

Comments

@aduverger
Copy link
Member

High-level design:
image

Overall logic:
Each time a json file is created or modified in the epd-raw-data-prod-eu-west-3 bucket, it sends an event to a SNS.
A SQS listens to this SNS and triggers a lambda that will read the json, and load its data into PostgreSQL.

UUID can be used for indexing the DB.
Maybe for v0 we can load everything into the same table?

Resources to create with cdk:

  • PostgreSQL (using aws-rds)
  • S3 Event notifications (modify existing S3 bucket)
  • SNS Topic
  • SQS + Dead-Letter Queue
  • Python Lambda to read json file and insert the data into PostgreSQL
@aduverger aduverger changed the title Load EcoPlaftform data for S3 to PostgreSQL Load EcoPlaftform data from S3 to PostgreSQL Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants