launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

This is my first Architecture Team email, so if I'm doing it wrong,
please let me know.  Francis suggested bringing these concerns to this
forum.

We have two systems that, broadly speaking, have the same function: the
job system and the build system.  They both exist to provide a way for
Launchpad to schedule and run processes, and gather their output.  You
might think that the build system's purpose is to build packages, but
Soyuz already has put in a bunch of work to generalize it, and it's now
able to perform "builds" for Rosetta that don't output a package.  This
is why it's under lp.buildmaster, not lp.soyuz.  (Though shouldn't it be
lp.services.buildmaster?)

If I had to characterize the differences between the build system and
the job system...

1. The build system always runs its code in a chroot, while the job
system runs it natively.  This means that the job system can have lower
startup costs, but cannot build packages for other distros.

2. The build system can run virtualized, which means it can safely run
untrusted code.

3. The build system distributes its work across multiple machines, using
a central buildmaster.  The Job system hasn't been tested on multiple
machines, but is intended to use the database to allow multiple machines
to coordinate.

I think that we should be working to converge these systems, so that we
can take advantage of their commonalities.  Both are proven, production
systems that should be modified carefully.  But we can start by
converging their data models.

I bring this up because Soyuz is currently in the middle of redesigning
the build system, and this is therefore a good opportunity to bring more
similarity to our systems.  The Code Import system is also a candidate
for convergence, but the Code team isn't currently redesigning it.

AIUI, part of the redesign of the build system involves creating a
Buildjob table with many fields that are similar to the Job table.  I
propose that the buildjob table should hold only fields that are not
generally relevant to jobs, plus a reference to the Job table.  This may
entail adjusting the Job table to better fit the build system's needs.

This will give us a common namespace for builds and jobs, so that we can
look them up the same way, and make it easier to display them together.
 It will also give us common representations and vocabulary for those
data that are common.

As time goes on, I expect the systems will come to depend on each other
more.  For example, when doing Recipe builds, we could catch problems
earlier by having an initial Job that ran the recipe, before handling
the actual package generation off to a Build.  Having overviews that
include both systems will become important.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvi4WsACgkQ0F+nu1YWqI03RACfYfmbTQujNvqPZvuDVzsY8caQ
GkwAmgOt8wnQ7I+Q21OceXRDjzvDaWR1
=NSi6
-----END PGP SIGNATURE-----