Which tools can we currently use in the DSH to run containers?

[arronlacey]

We are about to make use of the DSH for the EDoN Project and would like to use containers for our CI/CD pipelines in GitLab and store them in container registries for the researchers to pull down and use. Currently we had Docker in mind, especially as it has been rootless since 2019. But I'd like to know what is currently supported. Docker? Podman? Singularity (we might need some high performance, parallelizable containers). I saw that there was a discussion regarding podman but it doesn't seem like any decisions were made.

Any help would be much appreciated thanks!

[JimMadge]

Currently, none.

I would prefer not to use Docker because,

• Although it does support rootless containers, it requires extra configuration unlike Podman and Singularity/Apptainer
• The company is clearly moving towards more proprietary, subscription based software

I think Podman and Apptainer/Singularity make more sense. Podman should run an OCI container. Apptainer/Singularity can package OCI containers into single files than can be run like programs and should have performance benefits.

Podman and Apptainer support rootless containers out of the box (although you do need root to build Apptainer containers).

Maybe the bigger challenge here, though, is how do you review a container for ingress or egress?

[jemrobinson]

Under the data governance model we're using at Turing, reviewing a container would be the responsibility of the PI, data provider representative and referee. This shouldn't be too hard for ingress, but for egress it's very difficult to prove that the container doesn't include any of the sensitive data.

As Jim says, adding support for eg. Singularity is something we're considering, but our immediate roadmap is going to be driven by the requirements of the TRE reference architecture produced by the SATRE project so we can't be certain how high a priority this will be.

[arronlacey]

thanks @jemrobinson - For EDoN, at some point we will need to be thinking about egressing our containers out (it will be into another TRE run by Alzhiemer's Disease Data Initiative that non-EDoN external researchers can apply for access) so even though it's going to be difficult as you say, it's a longer-term goal for us.

[JimMadge]

Another thought. This sounds like a good use case for containers in a TRE. Being able to reproduce a pipeline (with all of its dependencies) inside and outside the TRE.

I could see a similar argument for building an 'app' as an apptainer container, testing it outside where development is more convenient before bringing it in for analysis on the sensitive data.

[arronlacey]

That sounds great to me - would certainly be easier to develop the container outside the TRE and pass into a 🤞painless🤞 ingress process to the TRE. At the moment we are building our containers via GitLab CI and storing in the container registry - so perhaps just ingress the entire Gitlab repo along with the container would be the way to go.

Do you see being able to make small edits to the container in the TRE and re-building in there possible too?

We are agnostic on which tool we use - Podman seems the best choice to me, but happy to go with how REG assesses what is appropriate.

[JimMadge]

Agreed, there are definitely advantages to enabling external development. @edwardchalstrey1's report also discusses this. Luckily, bringing things into a TRE feels less problematic than bringing things out.

Do you see being able to make small edits to the container in the TRE and re-building in there possible too?

I don't see any reason to block that. It might depend on the specific tools though. I believe Singularity/Apptainer requires root to build/modify container images so that may be a non-starter. It would also be difficult to bring those changes outside the environment because of the egress issues @jemrobinson spoke about.

[arronlacey]

Thanks Jim - it's all pointing to Podman it seems. I'll start creating some documentation and test repos for our users with Podman. 👍