Failure upgrading from 0.3.11 to 0.3.12

[rsignell-usgs]

We tried to follow the Qhub upgrade instructions to advance from 0.3.11 to 0.3.12 on a development deployment of Qhub (not the ESIP qhub).

After making the changes to 2 files (see screenshot below), the Github Action failed in "Render Changes to Template" step with:

Run qhub render -c qhub-config.yaml
Error: The schema validation of the qhub-config.yaml failed. The following error message may be helpful in determining what went wrong:

1 validation error for Main
default_images -> conda_store
  field required (type=value_error.missing)
Error: Process completed with exit code 1.

The changes we made that triggered the error are shown here:

What did we do wrong?

What does value_error.missing mean?
Is there a new parameter we need to include now?

We could provide access to the private repo if desired.

[tylerpotts]

@rsignell-usgs It looks like a new required value was added to the default_images key. Try adding the conda_store image as seen below:

default_images:
  jupyterhub: quansight/qhub-jupyterhub:v0.3.12
  jupyterlab: quansight/qhub-jupyterlab:v0.3.12
  dask_worker: quansight/qhub-dask-worker:v0.3.12
  dask_gateway: quansight/qhub-dask-gateway:v0.3.12
  conda_store: quansight/qhub-conda-store:v0.3.12

[tylerpotts]

The easiest way to get a look at an example of all the required keys is to run qhub init aws in an empty directory. That's what I did here to find the missing key

[rsignell-usgs]

@tylerpotts, okay, I fixed that, and also added the docker build for conda-store to the github actions:

But the conda-store image build is failing with:

Status: Downloaded newer image for quansight/conda-store:e2051a36e60bd3abd9aa44105f240b359ee6bab7
 ---> 5380e1b19c21
Step 2/2 : RUN apt-get update     && apt-get install -y     libgl1-mesa-glx     libegl1-mesa     libxrandr2     libxrandr2     libxss1     libxcursor1     libxcomposite1     libasound2     libxi6     libxtst6     build-essential     && rm -rf /apt/source/*
 ---> Running in 81f32c980a23
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Reading package lists...
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
The command '/bin/sh -c apt-get update     && apt-get install -y     libgl1-mesa-glx     libegl1-mesa     libxrandr2     libxrandr2     libxss1     libxcursor1     libxcomposite1     libasound2     libxi6     libxtst6     build-essential     && rm -rf /apt/source/*' returned a non-zero code: 100

[tylerpotts]

@rsignell-usgs I don't think you want to add conda-store to that part of the file. This particular github action is only if you want to custom-build the docker images, but the conda-store docker image already exists

[tylerpotts]

I also don't think Chris designed the conda-store image to be built via the github actions in this way. conda-store doesn't show up in either image.yaml or image-pr.yaml with a fresh initialization of a qhub directory.

[rsignell-usgs]

E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'

[tylerpotts]

Could you send me the full github URL? I tried https://github.com/esipfed/qhub-julia but that doesn't seem to be it

[rsignell-usgs]

https://github.com/rsignell-usgs/qhub-julia

[tylerpotts]

@rsignell-usgs Sad news... I went to that link and got a This invitation has expired. error

Would you mind sending that again please?

[rsignell-usgs]

Sent. Wow, you must have missed the expiration by mere hours!

[rsignell-usgs]

@tylerpotts were you able to gain access?
The only reason I was testing here was to do a dry run on the upgrade process, to see if I could just make changes via github (which would be ideal). If that's not possible, or it takes more effort than some other route, please let me know!

[rsignell-usgs]

Oh, and @dharhas tipped me off that 0.3.13 is now out.

I guess now my question is how best to upgrade from 0.3.11 to 0.3.13! 😄

[tylerpotts]

@rsignell-usgs (I deleted an earlier message which applied to a different cluster. If you see it in an email please ignore)

I'm seeing a similar error as you. I deployed my own AWS cluster with version 0.3.11 and attempted the upgrade to 0.3.13. I got a similar error as your latest one:

Error: Get "http://localhost/apis/storage.k8s.io/v1/storageclasses/nfs-mount-dev-share": dial tcp 127.0.0.1:80: connect: connection refused

It seems this is related to updating either the terraform or kubernetes version: terraform-aws-modules/terraform-aws-eks#1280

I tried a few of the different suggestions in various issues to no avail. I'll discuss this with the team to see if we can come up with some next steps.

[rsignell-usgs]

@tylerpotts , okay, thanks for trying!

[tylerpotts]

@rsignell-usgs Spoke with the team. At this point I think it's best to backup your data, deploy the new version, and restore the backup. Fixing this issue looks relatively complicated, and we're doing our best to optimize where we spend dev time.

I know it's going to be a pain to upgrade in this way. Let me know if you run into any snags and I'll be happy to help out.

[rsignell-usgs]

okay! Same backup procedure as before? (Log into storage pod, push tarball to s3?)

[tylerpotts]

@rsignell-usgs Yep that's right. Incidentally, I'm working on the new version of backups in #743

For now, the docs for backup are here: https://docs.qhub.dev/en/stable/source/admin_guide/backup.html?highlight=backup

[viniciusdc]

Hi @rsignell-usgs, sorry for the trouble during the update, has it succeeded following the fresh deployment?

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.