Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix export on "Nico's dash" #1091

Open
8 of 12 tasks
dirkcuys opened this issue Oct 26, 2022 · 2 comments
Open
8 of 12 tasks

Fix export on "Nico's dash" #1091

dirkcuys opened this issue Oct 26, 2022 · 2 comments

Comments

@dirkcuys
Copy link
Member

dirkcuys commented Oct 26, 2022

When exporting signups or learning circles the server times out. To prevent this, the export should be handled by an async celery task with the result being passed back to the user. Passing back the result could happen by either

  1. emailing the user
  2. implement a view for polling the status of the task.

For getting the file to the user the plan is to use AWS S3 and generate a pre-signed URL.

TODO (taking option 2 above and returning the pre-signed URL to the interface)

  • Add django-celery-result for storing task results in the db
  • Add view to query the status of the task
  • Move export logic to task
  • Upload exported file to S3
  • Generate pre-signed URL
  • Update UX on dashboard to poll for result and display link when ready
  • Update dashboard view to check for recent export tasks (to make sure you can still get the export if you refresh your page or click away)
  • Make sure to use a scope IAM user for uploading data to a private bucket and expire uploaded files + links in a short time, 30 minutes?
  • Implement for all export tasks (signup export, learning circle export, user export, ??)
  • Ignore results from other tasks
  • Consider shorter presigned URL that is generated by dynamic view on the server
  • Consider using lifecycle rules to expire objects
@dirkcuys
Copy link
Member Author

Could use S3 to host files. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

Workflow would be: create link at lc domain -> when user visits, generate a pre-signed s3 url and redirect user

@dirkcuys
Copy link
Member Author

Seems like django-celery-results doesn't create a db entry until celery returns a result :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Review
Development

No branches or pull requests

1 participant