yade-dev team mailing list archive

["Janek Kozicki (yade)" <jkozicki-yade@xxxxxxxxx>]

Hi All :)

I did an experimental switch to pybind, see this pipeline:

  https://gitlab.com/cosurgi/minieigen-real/pipelines/120246349

My observations about pybind11+minieigen:

1. pybind11 seems better organized and easier to use than boost::python

2. pybind11 has unstable API, what compiles on bullseye, does not
   compile on debian stretch:
       https://gitlab.com/cosurgi/minieigen-real/-/jobs/446852240
   (I only created debian-bullseye-pybind & debian-stretch-pybind
   images, failures on other linux distributions are because there is
   no pybind docker image)

3. compilation time without ccache (on my computer)
   base invocation:
   /usr/bin/time -f "\n\n\twall clock time: %E\n\n" make test_ci_mne_64 -j 1 CXXFLAGS="-D EIGEN_DONT_ALIGN -D EIGEN_DONT_VECTORIZE -D EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT"

   -O1
     boost::python make -j 10    :  31 seconds
     pybind             -j 10    :  30 seconds
     boost::python make -j 1     : 120 seconds
     pybind             -j 1     :  90 seconds

   -O3
     boost::python make -j 10    :  32 seconds
     pybind             -j 10    :  40 seconds
     boost::python make -j 1     : 140 seconds
     pybind             -j 1     : 150 seconds

   note: that's compilation time of minieigen only. Not yade ;) It is
   strange to observe that boost::python is easier to be optimized by
   compiler. Please try yourself. I only did two measurements without
   calculating standard deviation ;)

4. pybind does not segfault with Eigen vectorization enabled on older compiler.
   boost::python segfaults with Eigen vectorization on older compilers
   (We only very recently have added make_SSE to the pipeline, and it is
   with the newer compiler version). For someone who really wants SSE
   that's important. But in my opinion struggling for SSE serves no
   purpose without AlignedVector3, which isn't supported (yet).

5. both of them assume that minieigen sources are inside yade
   sources. The only difference is that the content of visitor.hpp is
   a little different e.g. replace py::extract<T> with py::cast<T>,
   and so on.

6. I don't know how many changes have to be done in entire yade code
   to switch to pybind.

7. pybind is more strict about mapping types C++ ↔ python, for pybind
   float is really float.
   For boost::python all three: float, double, long double were
   (partially without consistency) treated as double.
   for pybind a duplicate registration of an already registered type
   is a fatal error.

I understand that minieigen-src has no chance of getting into 20.04.

I conclude that we should put minieigen inside yade. If that will be
a boost::python version or a pybind version is yet to be decided.

best regards
Janek


Anton Gladky said:     (by the date of Sat, 22 Feb 2020 20:29:01 +0100)

> Hello Janek,
> 
> sorry for delays, "real" life is taking much more time when we get older :)
> 
> > * I would rather avoid pybind11 switch in a rush, better give it a few months
> 
> Maybe I was not clear, pybind11 is just a possible option. If we
> really decide to switch to it, we need to evaluate, whether it is
> suitable for us, whether it really helps to improve compilation times
> and what it costs for us (time).
> 
> > * I would prefer minieigen-src package (no extra compilation time for people who use `double`)
> > * minieigen-src in ubuntu 20.04 will let more people to test & use high precision.
> 
> Unfortunately, it is almost impossible for the time being. If I add a
> new "minieigen-src" binary package, the package should go through
> "NEW"-queue [1], which has almost 300 packages, waiting for review
> from Debian FTP-masters. It can delay upload for months.
> 
> We should probably consider alternative ways to help you to integrate
> high-precision stuff.
> 
> [1] https://ftp-master.debian.org/new.html
> 
> Best regards
> 
> Anton
> 
> Am Fr., 21. Feb. 2020 um 22:30 Uhr schrieb Janek Kozicki (yade)
> 
> <jkozicki-yade@xxxxxxxxx>:
> >
> > Hi Anton,
> >
> > Yes, the number of defines in Serialization.hpp is crazy.
> > Changing/removing them to use pybind11 would take a lot of work.
> > I'm sure it is possible, but not quickly.
> >
> > We can open an issue about pybind11 switch. Discuss pros and cons,
> > find a good strategy to deal with crazy defines and complete it later.
> >
> > Doing this switch right now is unrealistic. Working on another moving
> > target (pybind11) on top of an already moving target (minieigen HP
> > not fully integrated) is irresponsible. If possible I would prefer:
> >
> >   * minieigen-src package (slower compilation only for non-double)
> >   * or integrate minieigen with yade sources for now (rather undesirable
> >     due to compilation time).
> >
> > Whatever we do I would prefer to integrate HP, because doubles will
> > appear very quickly in the code.
> >
> > Ah, if we talked about pybind11 switch three months ago it would be
> > perfect timing for me :( In fact I was thinking about it that time,
> > but I didn't want to push into something that wasn't guaranteed to be
> > working [*]. Figuring out the correct way to convert high precision
> > to/from python with boost was difficult, for a while I prefer to not
> > repeat this task with pybind11. But surely we can get back to this
> > later.
> >
> > The extra time for compilation happens only when non-double type is used.
> > If minieigen-src package gets to 20.04 it will be possible later for
> > people to compile high precision yade from gitlab sources without big hassle.
> >
> > The build_focal_simple [1] is 5 minutes because it's not parallelized.
> > The problem is that pybuild is not accepting -j argument.
> > With make -j 6 it is faster [2][3].
> >
> > Also a fast quad_double (62 decimal places, package libqd-dev)
> > integration is in the works with boost::multiprecision developers [4].
> >
> > To summarize:
> >
> >  * I would rather avoid pybind11 switch in a rush, better give it a few months
> >  * I would prefer minieigen-src package (no extra compilation time for people who use `double`)
> >  * minieigen-src in ubuntu 20.04 will let more people to test & use high precision.
> >
> > I understand that this solution is temporary until we figure a
> > proper way to deal with pybind11. The 4 year LTS is more than enough
> > to solve pybind11 + Serialization.hpp problem.
> >
> > It is better to finish HP integration to not lose what is already
> > working.
> >
> > best regards
> > Janek
> >
> >
> > [*] pybind11 is a new dependency and a new package, boost is well
> >     integrated with itself, it was guaranteed to work.
> >
> > [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
> >
> > [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used this to test HP
> >
> > [3] In the pipeline with ccache it's usually not a problem. On
> >     debian build servers: it won't be a problem until we decide to
> >     make yade-float128 package. But if we made an accompanying
> >     minieigen-float128 this problem would disappear. But pybind11
> >     alternative is good also, just unrealistic right now.
> >
> > [4] https://github.com/boostorg/multiprecision/issues/184
> >
> >
> > Anton Gladky said:     (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
> >
> > > Hi Janek,
> > >
> > > we have misunderstanding here. python3-minieigen is the __binary__
> > > package and it is a bad idea to ship the source code with the package.
> > >
> > > Adding minieigen-src binary package is possible, but it looks like very
> > > undesired way.
> > >
> > > As I see, only Yade is using minieigen in the Debian. So, theoretically we
> > > can merge it again with Yade. Not sure, whether it is a good way though....
> > > Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
> > > and Gigabytes of used RAM.
> > >
> > > We should really have a look at pybind11 alternative, if it accelerates the
> > > compilation. But when I see Serialization.hpp, I am getting crazy from the
> > > number of "defines" there.
> > >
> > > Anton
> > >
> > > Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
> > > <jkozicki-yade@xxxxxxxxx>:
> > > >
> > > > Uh Anton, I'm sorry to be so boring:
> > > >
> > > > I have checked with sid in /etc/apt/sources.list:
> > > >
> > > >   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib
> > > >
> > > > and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
> > > >
> > > > There is no /usr/include/minieigen/*pp files inside :(
> > > >
> > > > For high precision to work, they are necessary. Maybe the proper way
> > > > to do this is to introduce python3-minieigen-dev package? I'm not
> > > > sure. These sources are needed because of these include [1] statements.
> > > >
> > > > I am attaching once again the file python3-minieigen.install which
> > > > installs *pp files. Even the *.cpp files are used. If you feel
> > > > that *.cpp files are too much, we could duplicate the .cpp files in
> > > > yade (they are rather short) and only include the *.hpp files (these
> > > > are quite long). But all *pp files in /usr/include/minieigen/ would
> > > > be perfect.
> > > >
> > > > Best regards
> > > > Janek
> >
> >
> > --
> > --
> > Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> > Gdańsk University of Technology
> > Faculty of Applied Physics and Mathematics
> > Department of Theoretical Physics and Quantum Information
> > --
> > http://yade-dem.org/
> > http://pg.edu.pl/jkozicki (click English flag on top right)


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)