yade-dev team mailing list archive

[Anton Gladky <gladky.anton@xxxxxxxxx>]

Hello Janek,

sorry for delays, "real" life is taking much more time when we get older :)

> * I would rather avoid pybind11 switch in a rush, better give it a few months

Maybe I was not clear, pybind11 is just a possible option. If we
really decide to switch to it, we need to evaluate, whether it is
suitable for us, whether it really helps to improve compilation times
and what it costs for us (time).

> * I would prefer minieigen-src package (no extra compilation time for people who use `double`)
> * minieigen-src in ubuntu 20.04 will let more people to test & use high precision.

Unfortunately, it is almost impossible for the time being. If I add a
new "minieigen-src" binary package, the package should go through
"NEW"-queue [1], which has almost 300 packages, waiting for review
from Debian FTP-masters. It can delay upload for months.

We should probably consider alternative ways to help you to integrate
high-precision stuff.

[1] https://ftp-master.debian.org/new.html

Best regards

Anton

Am Fr., 21. Feb. 2020 um 22:30 Uhr schrieb Janek Kozicki (yade)

<jkozicki-yade@xxxxxxxxx>:
>
> Hi Anton,
>
> Yes, the number of defines in Serialization.hpp is crazy.
> Changing/removing them to use pybind11 would take a lot of work.
> I'm sure it is possible, but not quickly.
>
> We can open an issue about pybind11 switch. Discuss pros and cons,
> find a good strategy to deal with crazy defines and complete it later.
>
> Doing this switch right now is unrealistic. Working on another moving
> target (pybind11) on top of an already moving target (minieigen HP
> not fully integrated) is irresponsible. If possible I would prefer:
>
>   * minieigen-src package (slower compilation only for non-double)
>   * or integrate minieigen with yade sources for now (rather undesirable
>     due to compilation time).
>
> Whatever we do I would prefer to integrate HP, because doubles will
> appear very quickly in the code.
>
> Ah, if we talked about pybind11 switch three months ago it would be
> perfect timing for me :( In fact I was thinking about it that time,
> but I didn't want to push into something that wasn't guaranteed to be
> working [*]. Figuring out the correct way to convert high precision
> to/from python with boost was difficult, for a while I prefer to not
> repeat this task with pybind11. But surely we can get back to this
> later.
>
> The extra time for compilation happens only when non-double type is used.
> If minieigen-src package gets to 20.04 it will be possible later for
> people to compile high precision yade from gitlab sources without big hassle.
>
> The build_focal_simple [1] is 5 minutes because it's not parallelized.
> The problem is that pybuild is not accepting -j argument.
> With make -j 6 it is faster [2][3].
>
> Also a fast quad_double (62 decimal places, package libqd-dev)
> integration is in the works with boost::multiprecision developers [4].
>
> To summarize:
>
>  * I would rather avoid pybind11 switch in a rush, better give it a few months
>  * I would prefer minieigen-src package (no extra compilation time for people who use `double`)
>  * minieigen-src in ubuntu 20.04 will let more people to test & use high precision.
>
> I understand that this solution is temporary until we figure a
> proper way to deal with pybind11. The 4 year LTS is more than enough
> to solve pybind11 + Serialization.hpp problem.
>
> It is better to finish HP integration to not lose what is already
> working.
>
> best regards
> Janek
>
>
> [*] pybind11 is a new dependency and a new package, boost is well
>     integrated with itself, it was guaranteed to work.
>
> [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
>
> [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used this to test HP
>
> [3] In the pipeline with ccache it's usually not a problem. On
>     debian build servers: it won't be a problem until we decide to
>     make yade-float128 package. But if we made an accompanying
>     minieigen-float128 this problem would disappear. But pybind11
>     alternative is good also, just unrealistic right now.
>
> [4] https://github.com/boostorg/multiprecision/issues/184
>
>
> Anton Gladky said:     (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
>
> > Hi Janek,
> >
> > we have misunderstanding here. python3-minieigen is the __binary__
> > package and it is a bad idea to ship the source code with the package.
> >
> > Adding minieigen-src binary package is possible, but it looks like very
> > undesired way.
> >
> > As I see, only Yade is using minieigen in the Debian. So, theoretically we
> > can merge it again with Yade. Not sure, whether it is a good way though....
> > Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
> > and Gigabytes of used RAM.
> >
> > We should really have a look at pybind11 alternative, if it accelerates the
> > compilation. But when I see Serialization.hpp, I am getting crazy from the
> > number of "defines" there.
> >
> > Anton
> >
> > Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
> > <jkozicki-yade@xxxxxxxxx>:
> > >
> > > Uh Anton, I'm sorry to be so boring:
> > >
> > > I have checked with sid in /etc/apt/sources.list:
> > >
> > >   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib
> > >
> > > and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
> > >
> > > There is no /usr/include/minieigen/*pp files inside :(
> > >
> > > For high precision to work, they are necessary. Maybe the proper way
> > > to do this is to introduce python3-minieigen-dev package? I'm not
> > > sure. These sources are needed because of these include [1] statements.
> > >
> > > I am attaching once again the file python3-minieigen.install which
> > > installs *pp files. Even the *.cpp files are used. If you feel
> > > that *.cpp files are too much, we could duplicate the .cpp files in
> > > yade (they are rather short) and only include the *.hpp files (these
> > > are quite long). But all *pp files in /usr/include/minieigen/ would
> > > be perfect.
> > >
> > > Best regards
> > > Janek
>
>
> --
> --
> Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> Gdańsk University of Technology
> Faculty of Applied Physics and Mathematics
> Department of Theoretical Physics and Quantum Information
> --
> http://yade-dem.org/
> http://pg.edu.pl/jkozicki (click English flag on top right)