[devseed: 4/6] [During Maintenance Window] Switch the VEDA prod hub to point to the EBS volume for storage

[sgibson91]

This task is to be completed during the maintenance window established in #4858. It tracks switching over the VEDA prod hub to use the EBS volume as in-cluster NFS for home dirs, instead of the EFS instance.

We should delete the public proxy service so that the hub is unreachable until we do a redeploy:

kubectl --namespace $HUB_NAME delete svc proxy-public

Quota enforcement

Set the quota to 10% + whatever is the current largest single user's home directory usage.

Tasks

• Perform one last migration: https://infrastructure.2i2c.org/howto/features/storage-quota/#migrating-existing-home-directories
  • Depending on how much time has passed between [devseed: 3/6] Preliminary migration of VEDA prod home dirs from EFS instance to EBS volume #4859 and this task, there'll be some updates to home dirs. We should do our best to copy them across now, and hopefully the process (using rsync) will be much faster as the bulk of the files have already been copied over.
• Establish the size of the largest home dir and add 10% for the quota: https://infrastructure.2i2c.org/howto/features/storage-quota/#enforcing-storage-quotas
• Configure the VEDA prod hub to use the EBS volume's IP address for NFS storage and redeploy: https://infrastructure.2i2c.org/howto/features/storage-quota/#switching-to-the-new-nfs-server

Definition of Done

• User servers can start
• User servers are connecting to the EBS volume for NFS storage
• Read/write functionality all works and file/dir ownership, permissions, and modified time (mtime) have been preserved and verified
• The proxy-public service has been brought back up by the redeploy