← Back to team overview

kicad-developers team mailing list archive

Re: Bug 1833851 - Massively Parallel Builds

 

Hi Seth,

I'm not a cmake or make expert so this may not fix anything but here is my
idea:
If make has issue with generated targets and can't merge them, how about we
give it a dummy non-generated target that has generated lexers as
dependency. Then all targets that need lexer files will have that dummy as
dependency instead of generated target itself.
Maybe that would give make a hint to build dummy target first and more
importantly only once?

Regards,
Andrew

On Sun, Jun 30, 2019 at 7:45 AM Seth Hillbrand <seth@xxxxxxxxxxxxx> wrote:

> Hi Devs-
>
> I'm stuck at a decision on this bug and would like some opinions from
> others.
>
> The background is the GNU make is not intelligent about parallel builds
> and doesn't merge generated targets.  So we need to serialize the
> dependencies but this breaks everytime we include the lexer header file
> in a different object or use the generated DSO for more than one target.
>   Eventually, we have sufficient parallel processes that the header gets
> generated twice.  If this happens at the same time, the build breaks.
>
> The two (ok, three) options I see are as follows:
>
> 1) Build the header file into a target directory instead of the source
> directory.  Then everyone gets to build their own with no conflict.  We
> end up building lots and lots of header/lexer files.  Developers won't
> see the files in the usual location.
>
> 2) We build the files like bitmap pngs.  That is, they are re-built by a
> different CMAKE flag and their header/cpp files are checked into the
> repository.  When we add tags to the parser, we need to explicitly
> regenerate the lexer files and check them in.
>
> 3) External scripting.  We could utilize an external python script to
> force one build at a time using lockfiles.  This would add python as a
> build dependency.  We could alternatively use flock in a shell script
> but I'm not sure that exists on all platforms.
>
> Does anyone have preference for which option we take?
>
> -Seth
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>

Follow ups

References