Setup Gitlab pipeline and provisioning on new Upstream project to enable two production infrastructures

[rija]

Pull request for issue:

• Create alternative infrastructure with live data copied #1957
• The tar function reports error in the bastion server #1990
• Install commonly use software to bastion server #1904
• Fix circular dependency between Gitlab pipeline and the Ansible bastion playbook run #1985
• Devise how to switch between current production and hot stand-by infrastructure #2009
• gigascience/security#4

This is a pull request for the following functionalities:

• Enable deployment of Gigadb on a new project in Upstream group as hot-standby alternative
• Make the main composer.json a regular editable file so automated dependency security checks can happen
• Add basic tooling to the bastion server like emacs and wget
• Alleviate's Yannan short timeout when ssh bastion server by installing tmux
• Fix issue where Yannan could not uncompress *.tar.bz2 archives by installing bzip2 and lbzip2
• Fix circular dependency between Gitlab pipeline and provisioning
• Document our plan for implementing blue/green deployment

How to test?

Deployment to your AWS staging environment

After checking out this PR's branch, pushing it to your Gitlab pipeline,
you can follow the "Setting up your Staging environment" section of the updated docs/SETUP_PROVISIONING.md document.

Note: the main change is the addition of a new playbook for database loading and CLI tools installation which fix the circular dependency problem.

Deploying to hot standby from Upstream's alt-gigadb-website pipeline

The hot standby infrastructure is already built, but you can test re-provisioning and deploying to it:

Follow the instructions specific to the upstream/alt-gigadb-website project in docs/sop/PROVISIONING_PRODUCTION.md for the staging environment.

Then follow the instructions in docs/RELEASE_PROCESS.md to create a fake release (choose a tag label that's obviously fake e.g: v00-rija-testing), that's going to be deployed live only to the upstream/alt-gigadb-website project, and not to our current production.

Once that release has been deployed to staging, you can resume the instructions in docs/sop/PROVISIONING_PRODUCTION.md from the "Provisioning live for upstream/alt-gigadb-website" section.

When guided to the blue/green deployment process, consider your fake release as a simple one (no infrastructure change and no database change): You can deploy to the Hot stand-by (currently the upstream/alt-gigadb-website) by following the instructions in "Deployment to a specific live environment" section fromdocs/sop/PROVISIONING_PRODUCTION.md

If everything goes well, you should be able to play with the new infrastructure:

1. When both pipelines are successful, navigate to the staging urls (check versions in footer, should match your fake release)

   1. https://staging.gigadb.org
   2. https://alt-staging.gigadb.org

2. Test you can connect to the bastion server for both staging environments with the centos user using the SSH key from the first two steps:

   1. [email protected]
   2. [email protected]

3. Test the deployment to live on Hot stand-by

   1. Navigate to https://alt-live.gigadb.org and check version in footer
   2. Connect with SSH to [email protected]
   3. Run the users_playbook.yml for the exact same username you already have on upstream/gigadb-website, you should then notice that you can ssh with that user using the same private key (because the public key is already in Gitlab variables)
   4. While connected to bastion.alt-live.gigadb.host, you can access /share/dropbox, and its content should be the same as what's on upstream/gigadb-website's EFS.
   5. While connected as centos user, exec crontab -l and notice that:
      • Database will be reset every day to be in sync with upstream/gigadb-website
      • It is not making and uploading backups to S3

blue/green deployment switchover

The last part of docs/sop/DEPLOYING_TO_PRODUCTION.md described the proposed plan for doing the blue/green deployment. Feel free to comment.

Changes to composer.json

composer.json is now a regular file, that's versioned and manually editable, so that automated dependencies security checks can be performed.
After checking out this branch, you should be able to execute ./up.sh as usual and everything should work as usual.

Addition of more basic tools

The new ops/infrastructure/data_cliapp_playbook.yml playbook will install:

• wget
• emacs
• bzip2 and lbzip2 (needed by the GNU tar command)
• tmux
• tree

On your AWS deployment of this branch (from the first section of the "how to test?") , you can check they work by executing the commands below in order on a bastion server:

tree .

tmux

wget https://gigadb.org

tar -cvjSf compressed.tar.bz2 index.html

rm index.html

tar -tf compressed.tar.bz2

tar -xf compressed.tar.bz2

rm compressed.tar.bz2

emacs index.html

C-x C-c to exit emacs

rm index.html

C-d to exit tmux
C-d to log off SSH

How have functionalities been implemented?

Blue/green deployment:

See the "Upstream projects" section of docs/sop/DEPLOYING_TO_PRODUCTION.md

Fixing the circular dependency issue:

Move all the bastion playbooks tasks that depend on the building and pulling of docker containers by the Gitlab pipeline into a new playbook ops/infrastructure/data_cliapp_playbook.yml which, unlike the other host configuration playbooks, is to be executed after running the Gitlab pipeline.

Any issues with implementation?

N/a

Any changes to automated tests?

N/a

Any changes to documentation?

• docs/sop/AWS_SETUP.md is replaced and augmented with docs/sop/DEPLOYING_TO_PRODUCTION.md
• docs/SETUP_PROVISIONING.md is updated to take into account the breaking down of the bastion playbook into two distinct playbook
• docs/RELEASE_PROCESS.md was moved to docs/sop/RELEASE_PROCESS.md and updated to reflect current practice and blue/green deployment changes
• docs/sop/PRODUCTION_DEPLOY.md was renamed docs/sop/PROVISIONING_PRODUCTION.md and updated to integrate two parallel infrastructures and blue/green deployment approach

Any technical debt repayment?

N/a

Any improvements to CI/CD pipeline?

The ops/infrastructure/bastion_playbook.yml was broken up for two reasons:

1. Bloat. Therefore, all the curators tools installation steps were moved to ops/infrastructure/data_cliapp_playbook.yml
2. Circular dependency between pipeline and provisioning: with this change, the steps laid out in docs/SETUP_PROVISIONING.md can be performed in order without ambiguity and clear boundaries.

[kencho51]

Hi @rija,

Since the PR #2049 has been reviewed and merged, and this PR has been rebased as well, happy to approve.