yade-dev team mailing list archive

[Anton Gladky <gladky.anton@xxxxxxxxx>]

Hi Bruno,

I have created several branches for all supported distributions (ubuntu and
debian).
Both (yade and yadedaily) are installed, but the doc-packages were dropped,
I
do not think they are needed for production.

Also I tried to reduce the size of the images, so they are just a little
more than 1gb.
Pipeline are scheduled for every day (after newly built yadedaily-packages
are placed
into the repository).

For yadedaily one needs to install also the python-mpi4py package. It
should be fixed with
this MR [1].

[1] https://gitlab.com/yade-dev/trunk/-/merge_requests/636

Please test the setup. And when it is OK, I will update the documentation.
Basically you need to run the following:

docker run --rm -it registry.gitlab.com/yade-dev/docker-prod:ubuntu20.04

or

docker run --rm -it registry.gitlab.com/yade-dev/docker-prod:debian-buster

Best regards

Anton


Am Fr., 5. März 2021 um 10:38 Uhr schrieb Bruno Chareyre <
bruno.chareyre@xxxxxxxxxxxxxxx>:

> Hi there,
>
> I'm planning to build new docker images in yade's gitlab for production,
> and possibly for development (see second part of this message, some
> background comes first).  This is open to suggestions.
>
> * Background:
>
> I recently started playing with "Singularity" images since I found our HPC
> department made it available on the clusters. There was also a user
> mentioning that on launchpad recently. From end-user POV, singularity
> images work like docker images, but a very practical difference is that it
> is allowed on our (and others') HPC. Docker isn't, for security reason.
>
> It made running yade so easy. The above command worked immediately, and
> should work just the same on every system with singularity installed:
> *ssh myHPC*
> *singularity exec
> docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily
> <http://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily>
> yadedaily --check*
>
> or equivalently:
>
>
>
>
> *export YADE='singularity exec
> docker://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily
> <http://registry.gitlab.com/bchareyre/docker-yade:ubuntu20.04-daily>
> yadedaily' $YADE --check $YADE myScript.py $ etc. *
>
> Key points:
> 1- singularity accepts docker images in input.
> 2- the above command is using some custom docker with yadedaily
> pre-installed (which then needs to be downloadable from somewhere where
> docker is permitted)
> 3- it is compatible with MPI(!). The host system's MPI is able to
> communicate with the image system's MPI in a scenario like this, as if it
> was just yade running natively on the host:
>
> *mpirun -np 500 $YADE someParrallelStuff.py *4- a condition for this MPI
> magic to work is that the mpi library is in the same version for the host
> and for the executed image
> 5- performance: no measurable difference compared to a yade compiled on
> the host (be it running -j1, -jN or mpiexec).
>
> For the moment the custom dockers are built in [1]
> <https://gitlab.com/bchareyre/docker-yade>.
> I'm also building a Singularity images with [2]
> <https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml>
> but I didn't really use it since I can build it from docker directly on the
> cluster (building the singularity image is implicit in *singularity exec
> docker://...*). Building on-site may not be allowed everywhere, though,
> and in that case [2] could be useful.
>
> * What can be done:
>
> I will move [1,2] or something similar to gitlab/yade-dev and advertise it
> in the install page. Also build more versions for people to use them. More
> versions because of the MPI point above (4): depending on the host system
> someone may want OMPI v1 (unbuntu16), or v2 (ubuntu18), etc.
>
> For production images it would make sense to just use canonical
> debian/ubuntu with yade and/or yadedaily preinstalled. But, it is not
> exactly what I did for the moment. Instead I used docker images from our
> registry. Which implies the images have yade, and also what it needs to
> compile yade (I didn't test compilation yet but it should work).
>
> I was thinking of splitting that into two types of images; minimal images
> for production and "dev" images with all compilation pre-requisites. Then I
> realized that the best "dev" image would be - by far - one reflecting the
> state of the system at the end of our current pipeline, i.e. one with a
> full /build folder and possibly ccache info (if not too large).
>
> If such dev images were pushed to yade registry then anyone could grab
> latest build and recompile incrementally. It could save a lot of
> (compilation) time for us when trying to debug something on multiple
> distros.
>
> And what about that?: compiling with a ubuntu20 docker image on a ubuntu20
> host should make it possible to use the pipeline's ccache while still
> running yade on the native system (provided that the install path is in the
> host filesystem).
>
> Maybe pushing to registry could be done directly as part of current
> pipeline, not sure yet. I am still thinking about some aspects but I think
> you get the general idea. Suggestions and advices are welcome. :)
>
> Chers
>
> Bruno
>
> [1] https://gitlab.com/bchareyre/docker-yade
>
> [2]
> https://gitlab.com/bchareyre/yade-singularity/-/blob/master/.gitlab-ci.yml
>
>
>
>
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~yade-dev
> Post to     : yade-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~yade-dev
> More help   : https://help.launchpad.net/ListHelp
>