launchpad-dev team mailing list archive

[Danilo Šegan <danilo@xxxxxxxxxxxxx>]

У чет, 05. 11 2009. у 12:17 +0000, Julian Edwards пише:
> I had a chat with many, many people yeterday and the upshot is that we'd like 
> to make the API into the Soyuz build system look like IJob, which is what the 
> Code folks use.
> 
> Aaron was kind enough to spend some time explaining IJob things to me 
> yesterday so I have a basic understanding of that now.
> 
> I will start by explaining how the current build farm schema works, which has 
> a large bearing on this new schema.
> 
> == Existing schema  ==
> 
> || Build || <build meta data> ... 
> || BuildQueue || Build (FK) | lastscore (int) | builder (FK) | logtail | ... 
> 
> The basic premise is that BuildQueue rows only exist as long as the job is 
> waiting to be run, or running.  As I've said previously, I want to retain this 
> behaviour so that the changes to Soyuz code are not as invasive.
> 
> Running jobs have the `builder` FK populated.
> 
> BuildQueue is sorted by "lastscore" and then ID.  This is important as build 
> jobs can be rescored and we rely on a pretty hairy query to fetch stuff from 
> the queue in the right order (there are other constraints, especially where 
> private builds are concerned).
> 
> == Proposed changes ==
> 
> The first stage is to retain the existing functionality for Soyuz package 
> builds, but allow other types of jobs to run later.
> 
> `Job` rows will be created in tandem with the build jobs, however log files 
> will continue to be stored in the librarian (they get big).
> 
> || BuildPackageJob || job (FK) | build (FK) | ... 
> || BuildQueue || job (FK) | job_type (enum) | lastscore (int) | ... 
> 
> BuildPackageJob will link the build record to the Job record.  BuildQueue 
> loses its build FK and now links to the build via the BuildPackageJob record 
> with the same job ID.
> 
> For other jobs we can add some more tables:
> 
> || BuildPackageBranchJob || job (FK) | sourcepackagebranch (FK) | ...
> || BuildPackageRecipeJob || job (FK) | ...
> || BuildTranslationsJob || job (FK) | ...	
> 
> There will be no doubt more clean-ups we can do in the Soyuz data/schema, but 
> that can wait for later.  For now, I just want to unblock everyone else.

Just to confirm, how do we expect to pass data for each job type to the
actual job?  I understand that it will live in the per-job-type-table,
but since each of these tables might contain different information, how
do we pass it on to builders (which have no DB access)?

I.e. for translations, assuming reasonable amount of decoupling in the
system, we'll need to pass at least three things around:

  * bzr branch URL/id (to be able to check it out)
  * relevant product + series or
  * relevant distribution + series + sourcepackagename

If we are going to be passing them as command line parameters, we might
as well put that directly into the generic Job table (as another string
field or existing json_data field?), and get rid of the intermediate
tables.

Cheers,
Danilo