launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

This got a bit long (parts of it should probably go into a spec).  I
hope it's still interesting.

Julian Edwards wrote:
> Howdy folks
> 
> Next week is UDS, which I am attending, so I won't be spending much time on 
> the generic build farm jobs work.
> 
> Muharem has been working on a branch to convert the schema (which jml approved 
> today) and to change our code that uses it to work with the new schema.  It is 
> largely done, and we hope to land it today.
> 
> https://code.edge.launchpad.net/~al-maisan/launchpad/bqim-479079
> 
> Once it has landed, this opens the door for some Translations and Code work.  
> Namely:
> 
>  * New tables, BuildPackageBranchJob, BuildPackageRecipeJob and 
> BuildTranslationsJob (or similar names as you prefer)

So, we finally get to think concretely about some things :-)

Let's start with a recipe.  In the abstract, a recipe is roughly a thing
that specifies how to combine branches into something that debuild could
turn into a source package -- a "debianized source tree".  Recipes today
are specified as text files that the 'bzr-builder' plugin parses and
acts on.  An obvious open question in choosing how to represent these is
either to store them in a parsed form, with all the structure of a
recipe in the schema or unparsed, as lumps of text.

Or, and there are actually reasons for doing this, we could do something
in between: store it mostly as text but replace the references to
branches in the text with references to database objects (probably the
id of entries in some RecipeBranch linking table).  This would let us
(a) check that the branches exist at parsing time (b) keep the
references up to date if the branch is moved or renamed (c) prevent
branches that are referenced in Recipes from being deleted.

Floating around in a very abstract way is the idea of recipe
inheritance.  I don't know how this works.

Separately, we need to decide where a recipe lives.  The current
thinking is
"https://launchpad.net/ubuntu/karmic/+source/some-package/+recipe/recipe-name";
which seems OK to me (we'd have to trust a bit that this recipe would
build a recipe for some-package in karmic, but that doesn't seem any
different to say branches today).

Finally, we could stick an archive on the recipe, but maybe we don't
want to.  I'll talk about this a bit more later in the mail.

This leads to a schema a bit like:

Recipe:
 - id, registrant, date_created, owner, date_last_modified
   - all standard launchpad fields.  the owner would be able to edit
     the recipe.
 - name
   - the last bit of the url
 - distroseries, sourcepackagename
   - provides the rest of the url
 - recipe
   - a text field containing the text of the recipe (probably with
mangled branch references so lp:foo would be replaced with lp:21435)

RecipeBranch:
 - id, recipe, branch
   - all obvious i hope

What follows hopefully doesn't depend too much on how the above gets
decided in the end.

For the job of building a recipe into a source package we'll have a
BuildSourcePackageFromRecipeJob table.  I foresee this table looking like:

BuildSourcePackageFromRecipeJob
 - job
 - recipe
 - archive?

BuildQueue will get a row with a job column will reference same job and
have a particular job_type.

One of the things bzr-builder does when it creates the debianised source
tree is create a manifest, which is a sort of frozen version of a recipe
-- it references particular revisions of the branches so as to allow a
repeat of exactly this build.  We could use a manifest like this to
actually run the recipe: at the point where the build is requested, we
make the manifest and stuff it into the database.  This seems like a
neat idea, but isn't how bzr-builder works now as far as I can tell.

This doesn't include anything that will actually create
BuildSourcePackageFromRecipeJob rows (say every day for a daily build
PPA).  I guess we can worry about this later.

I think the current plan is to use bzr-builder to make the debianized
source tree and bzr-builddeb to then make the source package.  I'm
presume the process for getting the source package off the builder and
into the process of being built will follow that of the existing
builders: the builder will tell the buildd-manager where to get the
.dsc, the manager will parse this to find the other parts of the package
and then grab them, shove all of the files into the librarian and
trigger the existing parts of soyuz to look at them somehow[1].

Something that's missing from all the above is how the archive is
selected.  It's more or less essential that the
BuildSourcePackageFromRecipeJob knows the archive, so that the generated
source package can be built for the right one.  It could be tied to the
recipe or it could be supplied when the job row is created.  In some
sense the archive is totally orthogonal to the recipe, but OTOH, I can't
really see the use case for targeting more than one archive with a
recipe.  Advice welcome.

In case the above wasn't enough, here's some things I haven't thought
hard about:

 - do people want to subscribe to a recipe?
   - does this mean getting notified when the recipe builds or fails to
     build?
   - does this mean getting notified when the recipe is changed?

 - the whole privacy thing.
   - do we only allow recipes to be created that reference branches the
     owner can see?
   - is having the people who can view the recipe being the intersection
     of those that can see the branches reasonable?
   - the issues of accessing private branches from the buildslaves
     scares me a bit, I hope we can avoid worrying about that until some
     time in 2010.

>  * Model code for new tables
> 
> The new tables should reference the Job table, other than that you are free to 
> have whatever data you need for this build job type.  I expect you'll need to 
> spend some time working this out.
> 
> The model code should implement the interface ISoyuzJob (although this is a 
> terrible name, it will be changed) which is declared in 
> lib/lp/soyuz/interfaces/soyuzjob.py.

This file doesn't seem to exist?

> See lib/lp/soyuz/model/buildpackagejob.py for an example of something that 
> implements it.
> 
> Tim and Danilo, I think you're putting mwh and jtv on these tasks?
> 
> Michael N, who is at UDS for the first 2 days, will start the refactoring work 
> on IBuilder when he gets back (Hi Michael!).  The current implementation is 
> very Soyuz-specific and references Builds and the like.  We need to separate 
> it into a basic IBuilder and more specific IPackageBuilder non-model class.  
> Once that is done, we can start writing IPackageBranchBuilder etc which will 
> deal with the specifics of handing off and receiving jobs of other types 
> to/from the build farm.  The underlying IBuilder should be doing most of this 
> work and it's up to Michael to figure out how it will work, exactly.

In coarse outline, building a source package from a recipe isn't very
different from building a binary from a source package, so this sounds
like it will be a mass of (presumably devilish) details rather than deep
design work.

> There is one nightmare part to this - the implementation of 
> IBuilder.findBuildCandidate().  This is currently a very ugly query that 
> decides which job to dispatch next, based on a few criteria about the jobs.  I 
> expect it to get to troll-like ugliness by the time we finish and I'm not sure 
> how it will work yet, it depends on how Michael re-factors IBuilder and sets 
> up an interface that other builders must implement.

Wow, that's a beauty alright.

> See more details here:
> https://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation
> which includes the project plan as a Planner attachment.

All comments most welcome.

Cheers,
mwh

[1] I guess the fact that these packages aren't signed will bite us in
   the ass somehow or other at some point, but I don't know how much it
   affects how this bit would work.  We don't *have* to get the source
   package files into Soyuz via the Librarian I guess.