Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move from dockerhub to quay.io (and consider we got an 800$ charge!) #688

Open
8 of 9 tasks
yuvipanda opened this issue Oct 18, 2023 · 55 comments
Open
8 of 9 tasks

Comments

@yuvipanda
Copy link
Collaborator

yuvipanda commented Oct 18, 2023

@choldgraf's card is apparently attached to the jupyterhub org on dockerhub, and it just got an 800$ charge!

image

This is not supposed to happen, as we have 'sponsored oss' status (I'm not sure who got that for us, but THANK YOU to whoever that was).

While I guess this charge is a mistake on their part, it at least feels sufficient motivation for me to move us completely over to quay.io. We already moved repo2docker (jupyterhub/repo2docker#1076 (comment)), and while I can't find the issue right now, the 'sponsored OSS' status was why we didn't move everything else. Time to move it now I think.

Repos to move

@sgibson91
Copy link
Member

sgibson91 commented Oct 18, 2023

@consideRatio
Copy link
Member

Was @choldgraf's card charged and is it now removed so it can't be charged going onards?

image

image

@minrk
Copy link
Member

minrk commented Oct 18, 2023

👍 to moving everything to quay and leaving docker. I don't think anything's been charged, but let's make sure we don't give them a chance to charge in the future.

@mathbunnyru
Copy link

mathbunnyru commented Oct 18, 2023

I see this charge on the jupyter DockerHub account as well (also unsuccessful), so we will also have to move it.

I think I understand what's going on.

https://www.docker.com/community/open-source/application/

Core contributors of your project namespace get access to a year-long Docker Team subscription

#559 (comment)

And, based on this message, it was exactly one year ago when the upgrade to Docker Team happened.
I didn't think that it would only last a year, and then Docker would charge us for the next year - doesn't really sound like a "Docker-Sponsored Open Source Program", more like "Docker Team One Year Trial".
I missed it a year ago, I'm so sorry.
Hope no one was charged (at least DockerHub says the payment failed, which is good).

@mathbunnyru
Copy link

👍 to moving the jupyter account to quay as well.

I will work on jupyter/docker-stacks move this or next week (I suppose it will be the most difficult repo to move, so it will take a bit of time).

@manics
Copy link
Member

manics commented Oct 18, 2023

Could we push the important images (mostly jupyterhub) to both docker hub and quay.io for a release cycle or two, but update all docs, config and example repos to use quay.io? That'll ensure anyone pulling/checking latest will still get updates in the short term.

@choldgraf
Copy link
Member

I've sent a support request to Docker asking them WTF is going on. I'll let you all know if I hear back. I definitely agree we should just leave the platform because this is too much hassle if there are alternatives that work just as well.

I'd suggest we:

  • Try to get one more year of service.
  • But plan to migrate off ASAP.
  • Use that year as buffer to give us some breathing room.
  • Cancel service as soon as we're migrated off.

@manics
Copy link
Member

manics commented Oct 18, 2023

I don't think we need another year of service (though there's no harm if someone wants to apply). It means people may hit the rate limits when pulling, but hopefully that means they'll look at the readme and see that they should move to quay.io

@yuvipanda
Copy link
Collaborator Author

yuvipanda commented Oct 20, 2023

I built a small tool that'll migrate some but not all tagged images across the registries! https://github.com/yuvipanda/move-off-dockerhub.

I've moved k8s-hub image's released versions only over to quay.io now, so if people do end up being throttled by dockerhub, they can simply point to quay.io and get the exact same image (as it's copied).

image

I'll do this for all the z2jh and chp images, and the JupyterHub ones as well.

@manics we can do this in reverse for a bit to dockerhub as well.

@mathbunnyru
Copy link

@yuvipanda please, change the email of the jupyterhub quay.io account to [email protected]. It will only change the icon for the organization (and the images).

@mathbunnyru
Copy link

mathbunnyru commented Oct 21, 2023

After 2 days of work with ~25 commits, I've finally switched http://github.com/jupyter/docker-stacks/ to Quay.io.
An example: https://quay.io/repository/jupyter/base-notebook
Only 1 of these 25 commits is actually renaming (mostly, sed-like), all the others are gonna stay if we ever decide to change the registry again.

Cool features work fine too - automatic registry description update, docker manifest to create multi-arch images, testing contributed recipes, and automatic wiki update.

Nice features of Quay.io I noticed:

  • the tags page visually shows which tags have the same content. As we push many tags for one image, this is useful
  • Security scan is built-in for free on the same page (remote Docker Scout is not free, as far as I know)
  • Having usage logs for free is nice as well
  • the tag history page shows when a tag was moved - it might be helpful during debugging (and lots of our tags are moving)

Things I don't like/don't work:

  • The first opening of Quay.io where you need to set which cookies you allow is a banner. It doesn't look/feel nice and is so slow.
  • I don't think there are badges with the number of stars/pulls/size of an image (currently, I use shields.io with Docker Hub values)

yuvipanda added a commit to yuvipanda/jupyterhub that referenced this issue Oct 22, 2023
See jupyterhub/team-compass#688
for context.

I've also added `QUAY_USERNAME` and `QUAY_PASSWORD` to environment
secrets, but *not* `env.REGISTRY`. I will do so once this gets
merged.
yuvipanda added a commit to yuvipanda/jupyterhub that referenced this issue Oct 22, 2023
See jupyterhub/team-compass#688
for context.

I've also added `QUAY_USERNAME` and `QUAY_PASSWORD` to environment
secrets, but *not* `env.REGISTRY`. I will do so once this gets
merged.
yuvipanda added a commit to yuvipanda/kubespawner that referenced this issue Oct 22, 2023
@choldgraf
Copy link
Member

I went ahead and applied for another year of Open Source account status with DockerHub, and got an e-mail that it was approved today. Could somebody confirm that we do in-fact have this status now?

@consideRatio
Copy link
Member

image

Not yet, but I recall that this wasn't reflected instantly last time either.

@manics
Copy link
Member

manics commented Oct 24, 2023

I received an email yesterday:

Hello jupyterhub!

We’re sad to see you go. This email confirms that your Docker Team - Annual subscription for your account has been canceled.

🤷🏽

@consideRatio
Copy link
Member

Dockerhub status looks like this:

image

I received this email:

image

@yuvipanda
Copy link
Collaborator Author

At least once the JupyterHub image moves are complete, we should go to all the READMEs in the jupyterhub dockerhub and write them out as redirects.

yuvipanda added a commit to yuvipanda/binderhub that referenced this issue Oct 26, 2023
I've also put in appropriate secrets for QUAY_USERNAME and
QUAY_PASSWORD

Ref jupyterhub/team-compass#688
yuvipanda added a commit to yuvipanda/dockerspawner that referenced this issue Oct 26, 2023
Bump up a couple other versions in the Dockerfile examples as
well.

Ref jupyterhub/team-compass#688
@yuvipanda
Copy link
Collaborator Author

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

@miraculixx
Copy link

miraculixx commented Nov 13, 2023

ok, everything has moved now! \o/

I appreciate the move to quay.io and everyone's effort to make this happen. I just came across this update by chance, noticing the reference to quay.io in the docs, and I'm wondering about the scope of the migration strategy.

Specifically I noticed that, as of writing this, only the images tagged with python-3.11 are available from the quay.io repo. For example, docker.io/jupyter/datascience-notebook:python-3.10.11 does not seem to be available on quay.io.

Is this intended? If yes, what is the policy going forward as to which Python versions will be built/provided on quay.io v.s. docker hub, or will the latter be shut down, that is all previous images would be removed? Thank you

@yuvipanda
Copy link
Collaborator Author

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

@mathbunnyru
Copy link

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

I have a few:

  1. We don’t have classical releases. And we build at least weekly, meaning we have much more images than other projects.
  2. Some of our images are quite large. I suppose combining with first issue it will be difficult to move all of them.
  3. Our current tags in Quay.io definitely clash with our previous tags with Docker Hub. We shouldn’t rewrite these tags with older images.
  4. There were lots of incorrect tags in DockerHub till 2022-07-05.
  5. Till that date our images were sometimes not inherited properly one from another. So, a new image was inherited from an old one, for example.

I didn’t want this mess to be present in Quay.io.

@miraculixx answering your questions.

I intentionally didn’t push old images to Quay.io, mostly for the reasons mentioned above.
All new updates will only be provided on Quay.io.
At the same time, we do not plan to shut down Docker Hub - it will definitely affect our users in a bad way and we don’t want that. At the same time I expect most of our users to move to Quay.io images in a few years, especially because of python eol.

@manics
Copy link
Member

manics commented Nov 14, 2023

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

@mathbunnyru
Copy link

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

I think that’s a good idea.

@mathbunnyru
Copy link

mathbunnyru commented Nov 15, 2023

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

@manics @yuvipanda @miraculixx done in jupyter/docker-stacks#2032
There is also a small workflow - so if we ever need to move something, we can reuse it.

In a few days I will update the table and the information about using Docker Hub for old images.

Update: done

@yuvipanda
Copy link
Collaborator Author

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

@mathbunnyru
Copy link

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

@yuvipanda done for 4 images in jupyterhub repo, please, take a look: jupyterhub/jupyterhub#4634

@manics
Copy link
Member

manics commented Nov 17, 2023

Did you intentionally omit the arm64 images when copying the old docker-stack images?

@mathbunnyru
Copy link

mathbunnyru commented Nov 17, 2023

Did you intentionally omit the arm64 images when copying the old docker-stack images?

Nope, that's a bug, thank you!

Do you maybe know an easy way to fix this?
Afaik, I can specify arch, when pulling, but only one, and that doesn't help much.

I know this solution:

  1. Pull aarch64-<tag> and x86_64-<tag> single arch images
  2. Push these images with only changing the registry
  3. Use docker manifest to merge these images and create one multi-platform image <tag>. The previous step is needed otherwise docker manifest tells something about security.

Update: Implemented this solution: jupyter/docker-stacks#2035
But I don't really like it - it doesn't handle a case where an image exists only for one arch

@manics
Copy link
Member

manics commented Nov 17, 2023

@yuvipanda suggested using skopeo copy --multi-arch all in jupyterhub/zero-to-jupyterhub-k8s#3254 (comment)

@mathbunnyru
Copy link

@manics thanks. I fixed the problem.

@consideRatio
Copy link
Member

consideRatio commented Nov 24, 2023

I'd like to publish to both DockerHub and Quay.io instead of switching over fully, is that okay?

Both of these container registries are free services under no obligation to stay functional and free for us to publish to and users to pull from, so it could make sense to publish images to more than one container registry to allow us to switch over if there are issues for some or all users.

Publishing to both seems like a kinder transition than to stop publishing to docker hub, as that doesn't break workflows like building and using a custom image for each release.

As an example, here is a PR for configurable-http-proxy that accomplishes this, its a minimal change that will only add the extra upload time.

image

@consideRatio
Copy link
Member

consideRatio commented Nov 24, 2023

If there are no objections, I'd like to help all repos publish to Docker Hub alongside Quay.io.

Repos to publish to both Docker Hub and Quay.io

@manics
Copy link
Member

manics commented Nov 24, 2023

I think it makes sense to publish JupyterHub and CHP to both since they're the base images that people use directly. Z2JH and perhaps BinderHub makes sense from the perspective of redundancy.

I don't think we should bother with repo2docker though since we made the switch a long time ago and we're beyond the point where people still expect to find it on Docker Hub.

I also don't see the point of pushing mybinder.org-deploy to both registries. It's our production deployment which we fully control, and although it's a good example of how to go about deploying a mybinder type service there's no expectation that it'll work for anyone else. The registry serves the purpose of being an internal container distribution service amongst the mybinder federation members which just happens to be public.

@consideRatio
Copy link
Member

👍 thanks for the feedback on this @manics and @minrk! There are now four open PRs opened to review about this.

@minrk
Copy link
Member

minrk commented Nov 24, 2023

I agree with @manics on most points. I don't really see the benefit for the helm chart images, since those aren't images that users use directly, the URLs are internal to the chart. So my inclination would be just the standalone images that haven't migrated long ago, (i.e. just jupyterhub and chp, I think).

@consideRatio
Copy link
Member

consideRatio commented Nov 24, 2023

I'd like to retain publishing to z2jh because the jupyterhub/k8s-hub image is not seldom built from, for example 2i2c is maintaining a self-managed image we bump to and rebuuild whenever we bump the chart. Its simple to migrate of course but still also easy to publish to both still.

Also, its a failsafe for the future. With charts relying on access to various images, if quay.io is down but we have published all images to docker hub as well, the fix is just a config update for all users with issues.


From 2i2c's meta chart depending on JupyterHub, but building its own custom hub pod image:
image

@yuvipanda
Copy link
Collaborator Author

yuvipanda commented Nov 25, 2023

+1 to what @minrk and @manics said. I think it will help move people away from dockerhub, towards quay.io. For people who are inheriting from one of the k8s images, to bump up version numbers they have to change the FROM line anyway to specify new tags, so I think it's fine that they have to put a quay.io on front.

I'm also personally kinda pissed off and disappointed at dockerhub, given this is like the 3rd or 4th time (since this) I've been part of a community effort scrambling to try to figure out if we need to move off it or not, and it eventually 'working out' that an imminent move is not exactly needed - feels a little bit like hostage taking, and I don't wanna do that again. So, from an ideological perspective, I would like us to maintain as minimal a presence on dockerhub as possible, and make sure that our primary registry is quay.io.

@manics
Copy link
Member

manics commented Nov 25, 2023

How about if we only push Z2JH releases to Docker Hub? I think that's a nice compromise between not supporting Docker Hub, whilst still allowing some flexibility and redundancy for admins.

@yuvipanda
Copy link
Collaborator Author

Given @consideRatio has already done the work in a bunch of PRs, I want to respect that. Perhaps what we can do is:

  1. Merge those, so we simultaneously publish to both dockerhub and quay.io
  2. Set a 1 year deadline - don't renew the dockerhub open source program
  3. Write a blog post about moving our primary host from dockerhub to quay.io, and recommend people move. Describe that in 1 year, there will be pull limits from dockerhub, and new images may not be published there. Describe how to move.
  4. In all public materials, refer to quay.io as our primary host, and link to Dockerhub as a 'secondary host' available as backup.
  5. Push to more secondary hosts before the end of the year as redundancy. I think this would mean GitHub to start with.
  6. Once the year is up, publish another blog post about how users should be using images on quay.io instead.

I think this eases users out, which I think is @consideRatio's original goal in making these PRs. I think any org that allows pulling from dockerhub will also allow pulling from quay.io (or GitHub registry), so I think the long term flexibility is not that limited.

How does this sound?

@minrk
Copy link
Member

minrk commented Nov 27, 2023

That all sounds great and sensible, thanks @consideRatio and @yuvipanda!

@consideRatio
Copy link
Member

consideRatio commented Nov 27, 2023

  1. Push to more secondary hosts before the end of the year as redundancy. I think this would mean GitHub to start with.

👍 for use GitHub's ghcr.io as the secondary container registry, I've already used it a bit in the dask org for dask-gateway, its worked out fine and has some nice features related to permission control for example. There are some relevant notes about this in this long post, copied here for future reference:

  • ghcr.io release preparation
  • BLOCKING UPSTREAM: It appears that the instructions at https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#inheriting-access-for-a-container-image-from-a-repository are outdated, and I've failed to manage to push a public image to ghcr.io. I will investigate this further.
    UPDATE: This was resolved, the UI had changed, but there was a "package settings" after all but a bit tricky to find.
  • Publish a dummy ghcr.io/dask/dask-gateway and ghcr.io/dask-gateway-server container manually so we can configure settings.
    1. You must be a Dask organization admin
    2. You must create a personal access token (PAT) for your account with write:packages permissions at https://github.com/settings/tokens/new.
    3. Login to the ghcr.io container registry with the PAT:
      docker login ghcr.io -u your-username
      
    4. Push a dummy image with a label to couple it with a github repo and enable us to configure permissions
      echo "FROM alpine" > Dockerfile
      docker build . --tag ghcr.io/dask/dask-gateway --label org.opencontainers.image.source=https://github.com/dask-gateway
      docker build . --tag ghcr.io/dask/dask-gateway-server --label org.opencontainers.image.source=https://github.com/dask-gateway-server
      docker push ghcr.io/dask/dask-gateway
      docker push ghcr.io/dask/dask-gateway-server
      
    5. Declare permissions/access for each container package
      For each container package: a) grant permissions to github actions to push to it, b) declare that access should be inherited, c) declare that it should be public. All settings should be found in the same page.

@yuvipanda
Copy link
Collaborator Author

I got this today:

image

No idea what that actually means.

@consideRatio
Copy link
Member

Sigh... I don't get whats going on, but maybe incorrectly triggered automation? So far, it seems we are sponsored still. The new "scout" plan seen below was something new i think.

Screenshot_20240120-135942

@manics
Copy link
Member

manics commented Jan 20, 2024

My feeling is we shouldn't spend too much time investigating this:

  • Keep promoting quay.io as the primary source of containers
  • Keep pushing to both Quay.io and Docker Hub where it's already configured and working
  • Move other repos from Docker Hub to quay.io if we want
  • If Docker Hub breaks at any point remove it rather than spending time trying to fix it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants