launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Muharem Hrnjadovic wrote:
> Michael Hudson wrote:
>> This got a bit long (parts of it should probably go into a spec).  I
>> hope it's still interesting.
> 
> Hello Michael,
> 
> thanks very much for writing this up. I have a few quick comments,
> please see below.
> 
>> Julian Edwards wrote:
>>> Howdy folks
>>>
>>> Next week is UDS, which I am attending, so I won't be spending much
>>> time on the generic build farm jobs work.
>>>
>>> Muharem has been working on a branch to convert the schema (which jml
>>> approved today) and to change our code that uses it to work with the
>>> new schema.  It is largely done, and we hope to land it today.
>>>
>>> https://code.edge.launchpad.net/~al-maisan/launchpad/bqim-479079
>>>
>>> Once it has landed, this opens the door for some Translations and
>>> Code work.  
>>> Namely:
>>>
>>>  * New tables, BuildPackageBranchJob, BuildPackageRecipeJob and 
>>> BuildTranslationsJob (or similar names as you prefer)
>> So, we finally get to think concretely about some things :-)
>>
>> Let's start with a recipe.  In the abstract, a recipe is roughly a thing
>> that specifies how to combine branches into something that debuild could
>> turn into a source package -- a "debianized source tree".  Recipes today
>> are specified as text files that the 'bzr-builder' plugin parses and
>> acts on.  An obvious open question in choosing how to represent these is
>> either to store them in a parsed form, with all the structure of a
>> recipe in the schema or unparsed, as lumps of text.
>>
>> Or, and there are actually reasons for doing this, we could do something
>> in between: store it mostly as text but replace the references to
>> branches in the text with references to database objects (probably the
>> id of entries in some RecipeBranch linking table).  This would let us
>> (a) check that the branches exist at parsing time (b) keep the
>> references up to date if the branch is moved or renamed (c) prevent
>> branches that are referenced in Recipes from being deleted.
> I like this.

Yay.

> How would we reference the linking table rows in the recipe
> text?

I see you figured this out.  In some sense I don't think it's very
interesting.

>> Floating around in a very abstract way is the idea of recipe
>> inheritance.  I don't know how this works.
>>
>> Separately, we need to decide where a recipe lives.  The current
>> thinking is
>> "https://launchpad.net/ubuntu/karmic/+source/some-package/+recipe/recipe-name";
>> which seems OK to me (we'd have to trust a bit that this recipe would
>> build a recipe for some-package in karmic, but that doesn't seem any
>> different to say branches today).
>>
>> Finally, we could stick an archive on the recipe, but maybe we don't
>> want to.  I'll talk about this a bit more later in the mail.
>>
>> This leads to a schema a bit like:
>>
>> Recipe:
>>  - id, registrant, date_created, owner, date_last_modified
>>    - all standard launchpad fields.  the owner would be able to edit
>>      the recipe.
>>  - name
>>    - the last bit of the url
>>  - distroseries, sourcepackagename
>>    - provides the rest of the url
>>  - recipe
>>    - a text field containing the text of the recipe (probably with
>> mangled branch references so lp:foo would be replaced with lp:21435)
>>
>> RecipeBranch:
>>  - id, recipe, branch
>>    - all obvious i hope
>>
>> What follows hopefully doesn't depend too much on how the above gets
>> decided in the end.
>>
>> For the job of building a recipe into a source package we'll have a
>> BuildSourcePackageFromRecipeJob table.  I foresee this table looking like:
>>
>> BuildSourcePackageFromRecipeJob
>>  - job
>>  - recipe
>>  - archive?
> Storing the archive here is a good idea IMHO. The question in my mind
> is: should this really be "BuildPackageFromRecipeJob" i.e. should it span
> both stages, the generation of the souce package and the subsequent
> "upload" and build from source package to a binary (.deb file)?
> 
> In that case we'd probably want a `Build` foreign key as well. It
> would also give us a more holistic view of the job. Whether that's
> desirable or cumbersome remains to be seen :)

Hm.  I was thinking that it would only build the source package, and
then the source package would get built by something like the existing
mechanism.  It doesn't have to be done this way, I guess, but it seems
sanest and easiest to me.

> [..]
> 
>>> The model code should implement the interface ISoyuzJob (although this is a 
>>> terrible name, it will be changed) which is declared in 
>>> lib/lp/soyuz/interfaces/soyuzjob.py.
>> This file doesn't seem to exist?
> 
> We renamed the interface and moved the files around, please see:
> 
> lib/lp/buildmaster/model/buildfarmjob.py
> lib/lp/buildmaster/interfaces/buildfarmjob.py

Ah.  That all looks OK (apart from BuildFarmJobType probably having too
many values; not a deep thing).

This is tangent, but how will scoring work?  I'm guessing the value
returned by IBuildFarmJob['score'] is stashed in the BuildQueue and then
it's the value in the BuildQueue that's examined by findBuildCandidates?

>>> See lib/lp/soyuz/model/buildpackagejob.py for an example of something that 
>>> implements it.
> 
> [..]
> 
>> [1] I guess the fact that these packages aren't signed will bite us in
>>    the ass somehow or other at some point, but I don't know how much it
>>    affects how this bit would work.  We don't *have* to get the source
>>    package files into Soyuz via the Librarian I guess.
> 
> I have a similar gut feeling :) The "upload" of a source package
> generated from a recipe occurs withing the confines of the data center
> but signing it with a "build farm" key would still be a good idea IMHO.

I guess it would help if I remembered how signing of binary packages
built by the current system worked...

Cheers,
mwh