Enable mechanism to upload new dataset from EFS to Wasabi and S3 Glacier at the same time

[pli888]

User story

As a curator
I want the dataset files I'm publishing to Wasabi to be also backed up to S3 Glacier
So that I have piece of mind that any dataset files we publish has a backup available

Acceptance criteria

Given there are new GigaDB datasets that are not backed up and not published
When I push the dataset files to Wasabi on the bastion server
Then the datasets files are saved to Wasabi bucket
And the dataset files are saved to AWS S3 Glacier class of storage

Additional Info

• 1) Create a S3 bucket for the backup of dataset files gigadb-datasetfiles-backup and use same subdirectory structure as on Wasabi main storage of dataset files #1964

• 2) Create a FilesBackupTool user in IAM and the create (and document) the appropriate IAM policy named AllowReadWriteBucketGigadbDatasetsFilesBackup for accessing the gigadb-datasetfiles-backup bucket #1965

• 3) Ensure rclone config for curators on the bastion server has configuration for both destination #1966

  • Wasabi (already existing)
  • S3 glacier

• 4) Create and deploy (using bastion_playbook.yml) a wrapper script that handles the transfer (rclone copy) and logging of selected dataset files to both destinations #1967

• 5) Add curators manual for operating the tool #1968

Note: the actual transfer already published dataset files is not part of this ticket, but should be dealt with in ticket #1963

Product Backlog Item Ready Checklist

• Business value is clearly articulated
• Item is understood enough by the IT team so it can make an informed decision as to whether it can complete this item
• Dependencies are identified and no external dependencies would block this item from being completed
• At the time of the scheduled sprint, the IT team has the appropriate composition to complete this item
• This item is estimated and small enough to comfortably be completed in one sprint
• Acceptance criteria are clear and testable
• Performance criteria, if any, are defined and testable
• The Scrum team understands how to demonstrate this item at the sprint review

Product Backlog Item Done Checklist

• Item(s) in increment pass all Acceptance Criteria
• Code is refactored to best practices and coding standards
• Documentation is updated as needed
• Data security has not been compromised (with particular reference to the personal information we hold in GigaDB)
• No deviation from the team technology stack and software architecture has been introduced
• The product is in a releasable state (i.e. the increment has not broken anything)

[pli888]

We have been advised to use another backup service. Ask @pli888 for details.

[rija]

Given there are new GigaDB datasets that are not backed up and not published
When I push the dataset files to Wasabi on the bastion server
Then the datasets files are saved to Wasabi bucket
And the dataset files are saved to AWS S3 Glacier class of storage

Hi @kencho51, @pli888,
@only1chunts mentioned this morning that that acceptance criteria that was our basis of thinking during sprint planning is not correct.
Because until a dataset is set to published, the curator may amend the files on Wasabi multiple times, so it's not worth backing up non-definitive files, multiple times, to a cold storage.

So I suggested Ken to update the transfer wrapper script to take flags --wasabi, --backup which will transfer the files to Wasabi or to S3 glacier depending on which flag is passed.

My only concern is that the backup is no longer automated, which is a big problem as humans have to remember to run transfer to backup.
So, I'll be creating a new backlog ticket for automatically asynchronously backup dataset files when the upload status is set to "Published". (Update: created #1992 )