← Back to team overview

kicad-developers team mailing list archive

Re: Bug 1833851 - Massively Parallel Builds

 

I see.
Exact same problem is solved with protobufs cmake module as it generates
headers as well. Maybe some inspiration can be taken from there.
https://github.com/Kitware/CMake/blob/master/Modules/FindProtobuf.cmake#L245

On Sun, Jun 30, 2019 at 12:40 PM Seth Hillbrand <seth@xxxxxxxxxxxxx> wrote:

> Thank you for the suggestion Andrew.  Unfortunately, one generated file is
> a header that gets included in source files. This means the target is built
> by any file that includes the header.
>
> -Seth
>
> On Jun 30, 2019 12:27 PM, Andrew Lutsenko <anlutsenko@xxxxxxxxx> wrote:
>
> Hi Seth,
>
> I'm not a cmake or make expert so this may not fix anything but here is my
> idea:
> If make has issue with generated targets and can't merge them, how about
> we give it a dummy non-generated target that has generated lexers as
> dependency. Then all targets that need lexer files will have that dummy as
> dependency instead of generated target itself.
> Maybe that would give make a hint to build dummy target first and more
> importantly only once?
>
> Regards,
> Andrew
>
> On Sun, Jun 30, 2019 at 7:45 AM Seth Hillbrand <seth@xxxxxxxxxxxxx> wrote:
>
> Hi Devs-
>
> I'm stuck at a decision on this bug and would like some opinions from
> others.
>
> The background is the GNU make is not intelligent about parallel builds
> and doesn't merge generated targets.  So we need to serialize the
> dependencies but this breaks everytime we include the lexer header file
> in a different object or use the generated DSO for more than one target.
>   Eventually, we have sufficient parallel processes that the header gets
> generated twice.  If this happens at the same time, the build breaks.
>
> The two (ok, three) options I see are as follows:
>
> 1) Build the header file into a target directory instead of the source
> directory.  Then everyone gets to build their own with no conflict.  We
> end up building lots and lots of header/lexer files.  Developers won't
> see the files in the usual location.
>
> 2) We build the files like bitmap pngs.  That is, they are re-built by a
> different CMAKE flag and their header/cpp files are checked into the
> repository.  When we add tags to the parser, we need to explicitly
> regenerate the lexer files and check them in.
>
> 3) External scripting.  We could utilize an external python script to
> force one build at a time using lockfiles.  This would add python as a
> build dependency.  We could alternatively use flock in a shell script
> but I'm not sure that exists on all platforms.
>
> Does anyone have preference for which option we take?
>
> -Seth
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
>

References