launchpad-dev team mailing list archive

[Bjorn Tillenius <bjorn@xxxxxxxxxxxxx>]

On Thu, May 06, 2010 at 11:34:03AM -0400, Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi all,
> 
> This is my first Architecture Team email, so if I'm doing it wrong,
> please let me know.

I think you're doing great.


> Francis suggested bringing these concerns to this
> forum.
> 
> We have two systems that, broadly speaking, have the same function: the
> job system and the build system.  They both exist to provide a way for
> Launchpad to schedule and run processes, and gather their output.  You
> might think that the build system's purpose is to build packages, but
> Soyuz already has put in a bunch of work to generalize it, and it's now
> able to perform "builds" for Rosetta that don't output a package.  This
> is why it's under lp.buildmaster, not lp.soyuz.  (Though shouldn't it be
> lp.services.buildmaster?)
> 
> If I had to characterize the differences between the build system and
> the job system...
> 
> 1. The build system always runs its code in a chroot, while the job
> system runs it natively.  This means that the job system can have lower
> startup costs, but cannot build packages for other distros.
> 
> 2. The build system can run virtualized, which means it can safely run
> untrusted code.
> 
> 3. The build system distributes its work across multiple machines, using
> a central buildmaster.  The Job system hasn't been tested on multiple
> machines, but is intended to use the database to allow multiple machines
> to coordinate.
> 
> I think that we should be working to converge these systems, so that we
> can take advantage of their commonalities.

This matches my impressions of the two systems as well, and I'm all for
unifying the two systems. I don't know how much we can unify, but we'll
find out.

> Both are proven, production
> systems that should be modified carefully.  But we can start by
> converging their data models.

Yes, that's a start. Another start would be to unify the APIs.
Personally I find it more useful to have consistent APIs, than
consistend data models, although they go a bit hand in hand. But I guess
my point is that it's ok for them to work a bit differently, as long as
the external APIs are consistent.

Both the job system and the build system are big and complicated
systems. I think it will be quite hard to just suggest things to be
modified one step at a time, without looking at how we want it to look
like in the end. At least for me :)

If Code and Soyuz people have time at UDS, I would love to see a
document creating, summarizing the important parts of the system,
explain how they can be unified (for example, giving example of common
methods), and list all the steps that need to happen.



-- 
Björn Tillenius | https://launchpad.net/~bjornt