← Back to team overview

yade-dev team mailing list archive

Docker/Singularity images for production (and possibly development)


Hi there,

I'm planning to build new docker images in yade's gitlab for production, and possibly for development (see second part of this message, some background comes first).  This is open to suggestions.

* Background:

I recently started playing with "Singularity" images since I found our HPC department made it available on the clusters. There was also a user mentioning that on launchpad recently. From end-user POV, singularity images work like docker images, but a very practical difference is that it is allowed on our (and others') HPC. Docker isn't, for security reason.

It made running yade so easy. The above command worked immediately, and should work just the same on every system with singularity installed:
/ssh myHPC//
//singularity exec docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily yadedaily --check/

or equivalently: /
///export YADE=//'singularity exec docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily yadedaily'
$YADE --check
$YADE myScript.py
$ etc.

Key points:
1- singularity accepts docker images in input.
2- the above command is using some custom docker with yadedaily pre-installed (which then needs to be downloadable from somewhere where docker is permitted) 3- it is compatible with MPI(!). The host system's MPI is able to communicate with the image system's MPI in a scenario like this, as if it was just yade running natively on the host:
/mpirun -np 500 $YADE someParrallelStuff.py
/4- a condition for this MPI magic to work is that the mpi library is in the same version for the host and for the executed image 5- performance: no measurable difference compared to a yade compiled on the host (be it running -j1, -jN or mpiexec).

For the moment the custom dockers are built in [1] <https://gitlab.com/bchareyre/docker-yade>. I'm also building a Singularity images with [2] <https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml> but I didn't really use it since I can build it from docker directly on the cluster (building the singularity image is implicit in /singularity exec docker://.../). Building on-site may not be allowed everywhere, though, and in that case [2] could be useful.

* What can be done:

I will move [1,2] or something similar to gitlab/yade-dev and advertise it in the install page. Also build more versions for people to use them. More versions because of the MPI point above (4): depending on the host system someone may want OMPI v1 (unbuntu16), or v2 (ubuntu18), etc.

For production images it would make sense to just use canonical debian/ubuntu with yade and/or yadedaily preinstalled. But, it is not exactly what I did for the moment. Instead I used docker images from our registry. Which implies the images have yade, and also what it needs to compile yade (I didn't test compilation yet but it should work).

I was thinking of splitting that into two types of images; minimal images for production and "dev" images with all compilation pre-requisites. Then I realized that the best "dev" image would be - by far - one reflecting the state of the system at the end of our current pipeline, i.e. one with a full /build folder and possibly ccache info (if not too large).

If such dev images were pushed to yade registry then anyone could grab latest build and recompile incrementally. It could save a lot of (compilation) time for us when trying to debug something on multiple distros.

And what about that?: compiling with a ubuntu20 docker image on a ubuntu20 host should make it possible to use the pipeline's ccache while still running yade on the native system (provided that the install path is in the host filesystem).

Maybe pushing to registry could be done directly as part of current pipeline, not sure yet. I am still thinking about some aspects but I think you get the general idea. Suggestions and advices are welcome. :)



[1] https://gitlab.com/bchareyre/docker-yade

[2] https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml





Follow ups