Ultralytics stopped working in the past hour

[AmoghBhagwat]

@omichel The ultralytics package started throwing errors around 1 hour ago, even though we have made no changes to the code. We are getting this error

File "/usr/local/webots-project/controllers/participant/yolov5/utils/general.py", line 38, in <module>
  from ultralytics.yolo.utils.checks import check_requirements
  ModuleNotFoundError: No module named 'ultralytics.yolo'

[omichel]

I have no idea on what is the problem. I believe this package is not part of the default Docker images of the contest. I guess you added it in your own Docker image?

[omichel]

Yes, if you asked for a specific game versus a specified opponent, it may be that the wrong controller was used. This is a known bug which shouldn't affect the finals anyhow.

Regarding CUDA, we have been regularly updating the NVIDIA and CUDA drivers (among other system libraries) automatically on the competition machine. We had problems this time because after some updates, the machine needs to be rebooted otherwise CUDA is simply broken. Rebooting the machine did fix the problem. I believe, that it is very unlikely that a proper CUDA update will break the behavior of the robot controllers as these CUDA updates are not backwards-compatibility-breaking updates but the normal updates that come with Ubuntu 22. However, I agree that it is still possible in theory, although very unlikely, that an update of any system library (including security updates) breaks the behavior of a controller. But you know, life is dangerous...

[AndrejOrsula]

Thank you, @omichel, for clarifying.

Yes, if you asked for a specific game versus a specified opponent, it may be that the wrong controller was used. This is a known bug which shouldn't affect the finals anyhow.

In this particular case, I did not request the game. The game was requested by another participant via a regular Trigger Evaluation CI run - I can tell it was a regular trigger because the participant that requested the game was matched against multiple consecutive opponents in the ladder. I am referring to the same CI run as above, during which the Docker image failed to build, and a controller from a different participant was used instead at the same time.

For completeness, I just rebuilt the Docker image of my submitted controller with --no-cache flag, and it was able to build without any issues. Therefore, I am still unsure why it would fail to build the Docker image on the server during the CI run. The layer that failed to build is there to install packages that need to be downloaded from online repositories. Therefore, a network issue on the client (CI runner that tried to build the image) or server (apt/pip repository from which download was attempted) is the only reason for a failure that I can think of - however unlikely that might be.

Would it be possible to check that all submitted Docker images are built without errors before using them in the finals?
If an image fails to build, the current cyberbotics/competition-record-action@v1 action only prints the following warning here, but in this particular case - it seems that the CI run just continued playing with some other participant's controllers instead of terminating the run. Edit: To clarify, a controller of some other participant was used instead of my controller for a match of my team. In this particular case, the controller of some other participant won the match against the opponent - but the win was counted as a win for my team.

Warning: Missing or misconfigured Dockerfile while building the opponent controller container

Checking whether docker build succeeds for all submissions should be enough.

---

Regarding CUDA, we have been regularly updating the NVIDIA and CUDA drivers (among other system libraries) automatically on the competition machine. ... I believe, that it is very unlikely that a proper CUDA update will break the behavior of the robot controllers as these CUDA updates are not backwards-compatibility-breaking updates but the normal updates that come with Ubuntu 22.

Oh, I see. Sorry, I misunderstood - I thought it was a manual update.

We had problems this time because after some updates, the machine needs to be rebooted otherwise CUDA is simply broken.

I have a feeling this indicates that the NVIDIA drivers were updated this time - as the NVIDIA drivers of the host enable NVIDIA Container Toolkit to use GPU inside containers (the driver consists of a kernel module, after all). Updating just CUDA libraries on the host should have no effect on Docker containers - as their local version of CUDA compiler/runtime inside the container would be used, right?

However, I agree that it is still possible in theory, although very unlikely, that an update of any system library (including security updates) breaks the behavior of a controller. But you know, life is dangerous...

Definitely, it is unlikely. I have had the "pleasure" of dealing with non-compatible NVIDIA driver versions within Docker containers before, but it was mostly due to older drivers of the target host. Hopefully, a simple update to a newer version is fine. 😅

Thank you again for everything.

[AndrejOrsula]

Unfortunately, I am afraid the aforementioned update might be causing my controller to fail due to an import error.

Participant BuzzBing triggered a run earlier today that contained a match against me. This time, my Docker image was built fine (I suppose it must have been the network issue in the previous run). However, I can see from the logs of the CI run that my controller crashed during initialization due to an import error in JAX (ML framework used by my controller).

CI run: https://github.com/cyberbotics/wrestling/actions/runs/6334709878/job/17204744036#step:6:1305

The error logs in the CI are truncated due to the line limit, but the error could be related to GPU - similar to the original error of this discussion.

If possible, @omichel, I would be able to provide a patch that changes the versions of Python dependencies used by my controller based on the current version of the NVIDIA drivers/CUDA. I would just need to know this version. The patch would consist of adding 1 extra layer to the Dockerfile that overrides the version of the ML framework (RUN pip3 install jax==X.Y.Z ...) - but no changes to the controller.

Please, let me know if this arrangement would be possible or if it would cause you too much trouble. Thank you in advance!

---

Here is a replay of a game where the controller worked. It was submitted ~15 hours before the deadline.

wrestling_potential_issue_with_ko_countdown_algo.mp4

Here is a replay from the game today, where the controller crashed due to an import error. The robot only moves forward with a pre-defined looping motion - but the actual controller (reinforcement learning agent) never gets initialized.

wrestling_controller_crashing.mp4

As mentioned above, I did not update my repository between these two runs.

[omichel]

Yes, I agree to make this change, as it is a bug that was not under your control at/after the deadline for submissions. Here is the CUDA version on the system:

$ nvidia-smi
Fri Sep 29 09:55:02 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 29%   32C    P8     6W / 120W |    264MiB /  3072MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1087      G   /usr/lib/xorg/Xorg                196MiB |
|    0   N/A  N/A      1356      G   /usr/bin/gnome-shell               55MiB |
|    0   N/A  N/A      2967      G   ...1/usr/lib/firefox/firefox        8MiB |
+-----------------------------------------------------------------------------+

[AndrejOrsula]

Thank you very much, @omichel! I really appreciate it and hope it doesn't cause you too much trouble.