Skip to content

Commit

Permalink
docs: connect provisioning and deployment with blue/green process
Browse files Browse the repository at this point in the history
  • Loading branch information
rija committed Aug 23, 2024
1 parent a3ed49e commit 08dbfa9
Show file tree
Hide file tree
Showing 2 changed files with 41 additions and 4 deletions.
9 changes: 8 additions & 1 deletion docs/sop/DEPLOYING_TO_PRODUCTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,9 @@ Logical Steps to perform for live deployment:
2. Turn off the data stores sync on the hot stand by side
3. Manually run S3backup with date+time suffix
4. On the hot standby, manually run `databaseReset <date+time>`
5. Deploy the release to hot stand-by
5. Deploy the release to hot stand-by by following the instructions in `docs/sop/PROVISIONING_PRODUCTION.md` in:
1. "Provisioning live for [...]" section, the one that matches the current Host stand-by
2. "Deployment to a specific live environment" section for the Hot stand-by
6. Have the business team to validate the hot standby deployment (Blue)
7. If validated, Swap the DNS records between the EIPs of the current production and the hot stand by
1. if not validated, see B.2 item below
Expand All @@ -294,6 +296,8 @@ Two possibilities from there:
A) Release on Blue is OK
1. Turn on the RDS S3 backup cronjob on Blue
2. Deploy the release to the former current-production, now the new hot stand-by (Green)
1. "Provisioning live for [...]" section, the one that matches the new Host stand-by
2. "Deployment to a specific live environment" section for the new Hot stand-by
3. Turn on EFS sync cronjob on Green
4. Turn on databaseReset cronjob on Green

Expand All @@ -312,3 +316,6 @@ and the changes are small and stateless.
In this case, there is no need to perform blue/green approach, and the process is almost similar to what we do now.
The only difference, is deployment needs to be done on both current production and hot stand by (starting with the latter)
to keep them in sync. Also, it is necessary to update the table at the top of this document with the new release versions.

1. "Deployment to a specific live environment" section from`docs/sop/PROVISIONING_PRODUCTION.md` for the Hot stand-by
2. "Deployment to a specific live environment" section from`docs/sop/PROVISIONING_PRODUCTION.md` for the Current Production
36 changes: 33 additions & 3 deletions docs/sop/PROVISIONING_PRODUCTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,9 +257,10 @@ need to be performed first on the Hot stand-by infrastructure. So that the Curre
by potential fallouts from deployment of that particular change.

That's why the next couple of sections that cover provisioning of live production environment for each Upstream projects
won't cover the deployment from Gitlab part.
For that, you will need to check the "Deploying releases" section of `docs/sop/DEPLOYING_TO_PRODUCTION.md`
to figure out which protocol to apply for the deployment part.
won't cover the deployment from Gitlab part.
Instead, we have an additional section call "Deployment to a specific live environment" which will need to be used
in conjunction with the "Deploying releases" section of `docs/sop/DEPLOYING_TO_PRODUCTION.md`
to figure out which protocol to apply that section with.

If the release includes changes to the infrastructure, then provisioning should be performed again, starting on the Hot Stand-by
infrastructure.
Expand Down Expand Up @@ -445,6 +446,35 @@ Provision files server:
env TF_KEY_NAME=private_ip OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ansible-playbook -i ../../inventories files_playbook.yml --extra-vars="gigadb_env=live"
```

### Deployment to a specific live environment

>**Important**: Only preform this section after checking the "Blue/green deployment of the release" of `docs/sop/DEPLOYING_TO_PRODUCTION.md`
Trigger a pipeline for the release tag you want to deploy on the suitable Gitlab project:

Go to either [Gitlab Upstream pipeline page](https://gitlab.com/gigascience/upstream/gigadb-website/-/pipelines)
or [Gitlab UpstreamAlt pipeline page](https://gitlab.com/gigascience/upstream/alt-gigadb-website/-/pipelines)
depending on which on you are deploying to
and run all the jobs in the staging build stage in your pipeline, and press "Run pipeline",
then select`develop` and confirm by clicking "Run pipeline".
This will execute all automated jobs.
There is a couple of manual jobs that will also need triggering: `PureFtpdBuildStaging` and then `PureFtpdDeployStaging`.

Since this is live production environment, you will need to build and deploy Tideways by running the manual jobs: `TidewaysBuildLive` and then `TidewaysDeployLive`

When the manual and automated jobs have all completed successfully, it will result in a partial deployment of GigaDB website to the infrastructure we've just provisioned.
Next, you can perform the last step which is to load the environment's database server with data and install the tools on bastion servers needed by the users:

```
env OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ansible-playbook -i ../../inventories data_cliapp_playbook.yml -e "gigadb_env=staging"
```

Finally, you need to warm the cache by following instructions from the wiki:
https://github.com/gigascience/gigadb-website/wiki/How-to-warm-the-in%E2%80%90memory-cache

The website should be visitable at https://alt-live.gigadb.host, and the bastion server ready at bastion.alt-live.gigadb.host if you were deploying the Hot Stand-by.
The website should be visitable at https://gigadb.org, and the bastion server ready at bastion.gigadb.host if you were deploying the Current Production.


## Additional features for executing ansible playbooks:
```
Expand Down

0 comments on commit 08dbfa9

Please sign in to comment.