launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Julian Edwards wrote:
> I had a chat with many, many people yeterday and the upshot is that we'd like 
> to make the API into the Soyuz build system look like IJob, which is what the 
> Code folks use.
> 
> Aaron was kind enough to spend some time explaining IJob things to me 
> yesterday so I have a basic understanding of that now.
> 
> I will start by explaining how the current build farm schema works, which has 
> a large bearing on this new schema.
> 
> == Existing schema  ==
> 
> || Build || <build meta data> ... 
> || BuildQueue || Build (FK) | lastscore (int) | builder (FK) | logtail | ... 
> 
> The basic premise is that BuildQueue rows only exist as long as the job is 
> waiting to be run, or running.  As I've said previously, I want to retain this 
> behaviour so that the changes to Soyuz code are not as invasive.
> 
> Running jobs have the `builder` FK populated.
> 
> BuildQueue is sorted by "lastscore" and then ID.  This is important as build 
> jobs can be rescored and we rely on a pretty hairy query to fetch stuff from 
> the queue in the right order (there are other constraints, especially where 
> private builds are concerned).
> 
> == Proposed changes ==
> 
> The first stage is to retain the existing functionality for Soyuz package 
> builds, but allow other types of jobs to run later.
> 
> `Job` rows will be created in tandem with the build jobs, however log files 
> will continue to be stored in the librarian (they get big).
> 
> || BuildPackageJob || job (FK) | build (FK) | ... 
> || BuildQueue || job (FK) | job_type (enum) | lastscore (int) | ... 
> 
> BuildPackageJob will link the build record to the Job record.  BuildQueue 
> loses its build FK and now links to the build via the BuildPackageJob record 
> with the same job ID.

This much, as amended by suggestions down thread, seems completely sane.

> For other jobs we can add some more tables:
> 
> || BuildPackageBranchJob || job (FK) | sourcepackagebranch (FK) | ...

I guess this isn't important _right now_ but will be important fairly
soon, and good to sort out asap, but the whole "build from branch" is
sadly not very well named -- it is looking like source packages will be
built from bzr-builder (https://edge.launchpad.net/bzr-builder) recipes
rather than branches.  The form in which recipes will be stored is not
yet fixed -- do we store them in a text column in a recipe, in some
fancy structured form, in text files in a database?  It would be good to
have a small pow-wow at UDS next week to talk about this (although I
won't be there myself).

Cheers,
mwh

> || BuildPackageRecipeJob || job (FK) | ...
> || BuildTranslationsJob || job (FK) | ...	
> 
> There will be no doubt more clean-ups we can do in the Soyuz data/schema, but 
> that can wait for later.  For now, I just want to unblock everyone else.
> 
> I welcome all comments on this schema, I want to start implementing it next 
> Monday.