yade-dev team mailing list archive

["Janek Kozicki (yade)" <jkozicki-yade@xxxxxxxxx>]

Hi,

I am not exactly sure what you want to discuss, so I will answer
some random parts of your email, and you will see that I completely
misunderstood :)

Do you want to create some sort of packages with yade installed inside?

Anton is building .deb packages for various distros.
Do you want to make some other kind of "packages" with yade, like
singularity_docker packages_images with yade inside?

I am not sure if yade-dev registry will be able to hold big
docker images. We may run out of space if we don't start paying
gitlab for hosting. For this reason Anton cannot build yade-debug.deb
packages, such packages are useful. But the files were too large. On
the other hand the .deb packages build by Anton are already hosted
off-gitlab, they are in http://www.yade-dem.org/packages/

Perhaps these singularity_docker packages should also be on yade-dem.org ?

> From end-user POV, singularity images work like docker images

The interesting stuff for me would be if we could use these HPC
singularity servers in our gitlab continuous integration pipeline :)
The yade-runner-01 quickly runs out of space whenever try to I enable
it ;-)

Maybe it is only a matter of single line in
file /etc/gitlab-runner/config.toml , change:

  executor = "docker"

to

  executor = "singularity"

I think this is quite likely.


> If such dev images were pushed to yade registry then anyone could grab 
> latest build and recompile incrementally. 

We already have incremental recompilation in our gitlab CI pipeline.
The ccache is used for that. The trick was to mount inside docker
(for you: inside singularity) a local directory from the host
filesystem, where the ccache files are stored.

The fact that it starts from "only" sources without the previously
compiled yade binaries (incremental compilation) changes nothing:
these binaries are in the ccache, and are quickly fetched from there.
In fact when everything is ccached the build step takes about 1 minute
(as you may have noticed :).

When we were configuring CI, Anton tried to always fetch ccached
files from gitlab. This worked the same in principle. Just the
download of these ccached binaries was taking over 10 minutes. This
is why we switched to mounting a local filesystem inside docker.

Note that the docker images are re-downloaded from gitlab only when
they have been rebuilt on https://gitlab.com/yade-dev/docker-yade/-/pipelines
And this download is pretty slow. Fortunately it happens only every
few weeks. Otherwise docker uses the cached linux distro image.

And this is why your planned singularity solution may not be efficient.

> It could save a lot of (compilation) time for us when trying to
> debug something on multiple distros.

Well, download once (wait for download to finish) then start working.
Not much difference to waiting for local compilation (for me that's
inside chroot, sometimes inside docker) then start working :)

> Maybe pushing to registry could be done directly as part of current 

pushing to registry is part of the pipeline on docker-yade:

https://gitlab.com/yade-dev/docker-yade/-/blob/master/.gitlab-ci.yml#L17


best regards
Janek

Bruno Chareyre said:     (by the date of Fri, 5 Mar 2021 10:37:59 +0100)

> Hi there,
> 
> I'm planning to build new docker images in yade's gitlab for production, 
> and possibly for development (see second part of this message, some 
> background comes first).  This is open to suggestions.
> 
> * Background:
> 
> I recently started playing with "Singularity" images since I found our 
> HPC department made it available on the clusters. There was also a user 
> mentioning that on launchpad recently. From end-user POV, singularity 
> images work like docker images, but a very practical difference is that 
> it is allowed on our (and others') HPC. Docker isn't, for security reason.
> 
> It made running yade so easy. The above command worked immediately, and 
> should work just the same on every system with singularity installed:
> /ssh myHPC//
> //singularity exec 
> docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily 
> yadedaily --check/
> 
> or equivalently: /
> ///export YADE=//'singularity exec 
> docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily 
> yadedaily'
> $YADE --check
> $YADE myScript.py
> $ etc.
> //
> 
> Key points:
> 1- singularity accepts docker images in input.
> 2- the above command is using some custom docker with yadedaily 
> pre-installed (which then needs to be downloadable from somewhere where 
> docker is permitted)
> 3- it is compatible with MPI(!). The host system's MPI is able to 
> communicate with the image system's MPI in a scenario like this, as if 
> it was just yade running natively on the host:
> /mpirun -np 500 $YADE someParrallelStuff.py
> /4- a condition for this MPI magic to work is that the mpi library is in 
> the same version for the host and for the executed image
> 5- performance: no measurable difference compared to a yade compiled on 
> the host (be it running -j1, -jN or mpiexec).
> 
> For the moment the custom dockers are built in [1] 
> <https://gitlab.com/bchareyre/docker-yade>.
> I'm also building a Singularity images with [2] 
> <https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml> 
> but I didn't really use it since I can build it from docker directly on 
> the cluster (building the singularity image is implicit in /singularity 
> exec docker://.../). Building on-site may not be allowed everywhere, 
> though, and in that case [2] could be useful.
> 
> * What can be done:
> 
> I will move [1,2] or something similar to gitlab/yade-dev and advertise 
> it in the install page. Also build more versions for people to use them. 
> More versions because of the MPI point above (4): depending on the host 
> system someone may want OMPI v1 (unbuntu16), or v2 (ubuntu18), etc.
> 
> For production images it would make sense to just use canonical 
> debian/ubuntu with yade and/or yadedaily preinstalled. But, it is not 
> exactly what I did for the moment. Instead I used docker images from our 
> registry. Which implies the images have yade, and also what it needs to 
> compile yade (I didn't test compilation yet but it should work).
> 
> I was thinking of splitting that into two types of images; minimal 
> images for production and "dev" images with all compilation 
> pre-requisites. Then I realized that the best "dev" image would be - by 
> far - one reflecting the state of the system at the end of our current 
> pipeline, i.e. one with a full /build folder and possibly ccache info 
> (if not too large).
> 
> If such dev images were pushed to yade registry then anyone could grab 
> latest build and recompile incrementally. It could save a lot of 
> (compilation) time for us when trying to debug something on multiple 
> distros.
> 
> And what about that?: compiling with a ubuntu20 docker image on a 
> ubuntu20 host should make it possible to use the pipeline's ccache while 
> still running yade on the native system (provided that the install path 
> is in the host filesystem).
> 
> Maybe pushing to registry could be done directly as part of current 
> pipeline, not sure yet. I am still thinking about some aspects but I 
> think you get the general idea. Suggestions and advices are welcome. :)
> 
> Chers
> 
> Bruno
> 
> [1] https://gitlab.com/bchareyre/docker-yade
> 
> [2] 
> https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)