Error to start docker service using older image

[bayuaji732]

What docker image(s) are you using?

tensorflow-notebook

Host OS system

Centos 7

Host architecture

x86_64

What Docker command are you running?

docker service create --replicas 1 --publish 16016:8888 --mount type=bind,source=/home/apps/certs,target=/etc/ssl/notebook --mount type=volume,source=jupyter-data,target=/home/jovyan/work --env JUPYTER_TOKEN=easy --network my_network --name jupyter jupyter/tensorflow-notebook:python-3.7 start-notebook.py --ServerApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --ServerApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

How to Reproduce the problem?

I ran that Docker service command, but the process stuck and the service was not started.
So I stopped it because it was still running endlessly.
But if I use jupyter/tensorflow-notebook:latest, it's running perfectly.

Command output

suf992acy53ennwyx8xoywhdt
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]
^COperation continuing in background.
Use `docker service ps suf992acy53ennwyx8xoywhdt` to check progress.

Expected behavior

I wish I could run a Docker service with SSL and an older version of Docker. I need to use Python 3.7.

Actual behavior

I ran that Docker service command, but the process stuck and the service was not started.

Anything else?

No response

Latest Docker version

• I've updated my Docker version to the latest available, and the issue persists

[mathbunnyru]

Could you please make your example reproducible?
Other developers should be able to easily reproduce your setup to be able to help you.

[mathbunnyru]

Also, Python 3.7 has reached EOL, so it is very unlikely that we will be able to rebuild/fix the image (if there is actually something wrong with the image). But maybe there is a way to make an existing image work for you.
But, I highly recommend you not to use EOL versions of python.
For example, Conda support 3.8-3.11

[bayuaji732]

Could you please make your example reproducible? Other developers should be able to easily reproduce your setup to be able to help you.

Sorry my bad.

1. Pull the image
   docker pull jupyter/tensorflow-notebook:python-3.7

2. Run the following command
   docker service create --replicas 1 --publish 16016:8888 --mount type=bind,source=/home/apps/certs,target=/etc/ssl/notebook --mount type=volume,source=jupyter-data,target=/home/jovyan/work --env JUPYTER_TOKEN=easy --network my_network --name jupyter jupyter/tensorflow-notebook:python-3.7 start-notebook.py --ServerApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --ServerApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

[bayuaji732]

Also, Python 3.7 has reached EOL, so it is very unlikely that we will be able to rebuild/fix the image (if there is actually something wrong with the image). But maybe there is a way to make an existing image work for you. But, I highly recommend you not to use EOL versions of python. For example, Conda support 3.8-3.11

I am following your suggestion for using newer python images:

1. jupyter/tensorflow-notebook:python-3.8 -> (same stuck docker service create)
2. jupyter/tensorflow-notebook:python-3.10 -> (same stuck docker service create)
3. jupyter/tensorflow-notebook:python-3.11 -> (success)

I think the problem is when I use an old version of the image. But actually I still need images with Python version 3.7 because of my application dependencies. If my memory serves me correctly, previously, the Docker stack provided code to create custom images. But now I can't find it. Can you show me how to create a custom image? Thank You

[mathbunnyru]

Could you please make your example reproducible? Other developers should be able to easily reproduce your setup to be able to help you.

Sorry my bad.

1. Pull the image
   docker pull jupyter/tensorflow-notebook:python-3.7
2. Run the following command
   docker service create --replicas 1 --publish 16016:8888 --mount type=bind,source=/home/apps/certs,target=/etc/ssl/notebook --mount type=volume,source=jupyter-data,target=/home/jovyan/work --env JUPYTER_TOKEN=easy --network my_network --name jupyter jupyter/tensorflow-notebook:python-3.7 start-notebook.py --ServerApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --ServerApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

I mean more like - how do you create your certificates in /home/apps/certs, do they cause the problem?
What is located in your jupyter-data folder? Have you tried not to mount this dir?
By reproducible I mean you should include all the steps so we have similar files (not necessarily the same).
Also, it's good to create a minimal example - to get rid of all the options which don't change the behavior. It sometimes includes some options/mounting and so on.

[mathbunnyru]

But I guess I see the problem here. You're using start-notebook.py.
You should use start-notebook.sh for older images (we kept it for new images as well, though, but with a warning).
Python version has been recently added, so it won't work in old images: #2006

Btw, if you created a minimal example, and stripped out all the parts that don't fix the command, you would be left with a very simple line and the problem would be easier to see.

[bayuaji732]

I mean more like - how do you create your certificates in /home/apps/certs, do they cause the problem? What is located in your jupyter-data folder? Have you tried not to mount this dir? By reproducible I mean you should include all the steps so we have similar files (not necessarily the same). Also, it's good to create a minimal example - to get rid of all the options which don't change the behavior. It sometimes includes some options/mounting and so on.

Oh sorry. I got the certificates from a third party, so I don't know how they created them. If you are okay, I can send it personally to you via email to check.
I already tried not to mount the jupyter_data volume, but it was still the same. Actually, jupyter_data is a fresh new volume.
And when I try the minimal example without the cert SSL command, it works. So I think the problem is that my SSL certs does not work with older versions of images.

[bayuaji732]

But I guess I see the problem here. You're using start-notebook.py. You should use start-notebook.sh for older images (we kept it for new images as well, though, but with a warning). Python version has been recently added, so it won't work in old images: #2006

Btw, if you created a minimal example, and stripped out all the parts that don't fix the command, you would be left with a very simple line and the problem would be easier to see.

I forgot to mention it, actually I already try both of the command:

1. start-notebook.py --ServerApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --ServerApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key
2. start-notebook.sh --NotebookApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --NotebookApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

here the output:
[email protected] | [I 2023-12-03 04:12:24.004 ServerApp] Serving notebooks from local directory: /home/jovyan
[email protected] | [W 2023-12-03 04:11:40.536 ServerApp] SSL Error on 9 ('127.0.0.1', 34140): [SSL: HTTP_REQUEST] http request (_ssl.c:1091)
[email protected] | [I 2023-12-03 04:12:24.004 ServerApp] Jupyter Server 1.19.1 is running at:
[email protected] | [W 2023-12-03 04:11:55.675 ServerApp] SSL Error on 9 ('127.0.0.1', 34278): [SSL: HTTP_REQUEST] http request (_ssl.c:1091)
[email protected] | [I 2023-12-03 04:12:24.004 ServerApp] https://651fe6e0a0ae:8888/lab?token=...
[email protected] | [W 2023-12-03 04:12:10.910 ServerApp] SSL Error on 9 ('127.0.0.1', 34350): [SSL: HTTP_REQUEST] http request (_ssl.c:1091)
[email protected] | [C 2023-12-03 04:12:13.086 ServerApp] received signal 15, stopping
[email protected] | [I 2023-12-03 04:12:24.004 ServerApp] or https://127.0.0.1:8888/lab?token=...
[email protected] | [I 2023-12-03 04:12:13.087 ServerApp] Shutting down 3 extensions
[email protected] | [I 2023-12-03 04:12:13.088 ServerApp] Shutting down 0 terminals
[email protected] | [I 2023-12-03 04:12:24.004 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[mathbunnyru]

I mean more like - how do you create your certificates in /home/apps/certs, do they cause the problem? What is located in your jupyter-data folder? Have you tried not to mount this dir? By reproducible I mean you should include all the steps so we have similar files (not necessarily the same). Also, it's good to create a minimal example - to get rid of all the options which don't change the behavior. It sometimes includes some options/mounting and so on.

Oh sorry. I got the certificates from a third party, so I don't know how they created them. If you are okay, I can send it personally to you via email to check. I already tried not to mount the jupyter_data volume, but it was still the same. Actually, jupyter_data is a fresh new volume. And when I try the minimal example without the cert SSL command, it works. So I think the problem is that my SSL certs does not work with older versions of images.

I am not very good with certificates and how they work.
I only have one idea how this might be fixed (and I'm not sure at all it's gonna work).
I assume the certificates you want to use are not supported by the certificates in the image.

What you need to do is to:

1. Update ca-certificates in the image (see below).
2. Set -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt when using docker run (or in your case docker service). This will make Python use system certificates, not the built-in ones. https://stackoverflow.com/a/42982144/4881441

This is how you build an image, you create a Dockerfile:

FROM jupyter/tensorflow-notebook:python-3.7
USER root
RUN apt-get update --yes && \
    apt-get install --upgrade --yes ca-certificates && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
USER ${NB_UID}

Then you run docker build -t my-tensorflow .
And then you use this image instead of jupyter/tensorflow-notebook:python-3.7.

You can send me your certs (use my github nickname + @gmail.com), I promise not to share/use them (only for testing the issue) and delete them when I'm done.

[bayuaji732]

I am not very good with certificates and how they work. I only have one idea how this might be fixed (and I'm not sure at all it's gonna work). I assume the certificates you want to use are not supported by the certificates in the image.

What you need to do is to:

1. Update ca-certificates in the image (see below).
2. Set -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt when using docker run (or in your case docker service). This will make Python use system certificates, not the built-in ones. https://stackoverflow.com/a/42982144/4881441

This is how you build an image, you create a Dockerfile:

FROM jupyter/tensorflow-notebook:python-3.7
USER root
RUN apt-get update --yes && \
    apt-get install --upgrade --yes ca-certificates && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
USER ${NB_UID}

Then you run docker build -t my-tensorflow . And then you use this image instead of jupyter/tensorflow-notebook:python-3.7.

You can send me your certs (use my github nickname + @gmail.com), I promise not to share/use them (only for testing the issue) and delete them when I'm done.

I am following your suggestion and run the docker like this:
docker service create --replicas 1 --publish 16016:8888 --mount type=bind,source=/home/apps/certs,target=/etc/ssl/notebook -e REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt --env JUPYTER_TOKEN=easy --network my_network --name jupyter my-tensorflow:latest start-notebook.sh --ServerApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --ServerApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

but still the same.

I am already sending my certificate to your email. I hope it can help you to reproduce my error.
Thank You

[mathbunnyru]

@bayuaji732 can you reproduce this with docker run?
I am not familiar with docker service create at all.

[mathbunnyru]

docker run -it --rm --publish 16016:8888 --mount type=bind,source=/Users/asalikhov/Downloads/certs,target=/etc/ssl/notebook --env JUPYTER_TOKEN=easy --name jupyter jupyter/base-notebook:python-3.7 start-notebook.sh --NotebookApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --NotebookApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

This works for me just fine - removed mounted volume, network, used docker run, also used base-notebook (I'm on aarch64).

[mathbunnyru]

Definitely need someone who understands how docker service create works.

[bayuaji732]

docker run -it --rm --publish 16016:8888 --mount type=bind,source=/Users/asalikhov/Downloads/certs,target=/etc/ssl/notebook --env JUPYTER_TOKEN=easy --name jupyter jupyter/base-notebook:python-3.7 start-notebook.sh --NotebookApp.certfile=/etc/ssl/notebook/ssl-bundle.crt --NotebookApp.keyfile=/etc/ssl/notebook/wildcard-ui_ac_id_2023.key

This works for me just fine - removed mounted volume, network, used docker run, also used base-notebook (I'm on aarch64).

Yes, you are right. using docker run running perfectly fine.

Definitely need someone who understands how docker service create works.

Yes, maybe I will use newest python version for temporary solution.
You can close this issue. I think the error in docker service -_-
Thank you very much for helping me. I hope you are healthy and successful

Summary:

1. Docker run with custom SSL using older version of images --> success
2. Docker run with custom SSL using newest version of images --> success
3. Docker service with custom SSL using older version of images (<= python 3.10) --> error
4. Docker service with custom SSL using newest version of images (>= python 3.11) --> success

[mathbunnyru]

Yes, maybe I will use newest python version for temporary solution.
You can close this issue. I think the error in docker service -_-

I will keep the issue open for a week - in case someone knows what is actually going on and then close.
Sorry I couldn't fix it.

[mathbunnyru]

Closing, as discussed earlier. If someone has any ideas, feel free to share them.