launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Julian Edwards wrote:
> Michael Hudson wrote:
>  > So, we finally get to think concretely about some things :-)
> 
> I hope it was worth the wait.

Well, it might have been better to have done this thinking last week in
Mooloolaba, but oh well...

>> Or, and there are actually reasons for doing this, we could do something
>> in between: store it mostly as text but replace the references to
>> branches in the text with references to database objects (probably the
>> id of entries in some RecipeBranch linking table).  This would let us
>> (a) check that the branches exist at parsing time (b) keep the
>> references up to date if the branch is moved or renamed (c) prevent
>> branches that are referenced in Recipes from being deleted.
> 
> This seems very reasonable to me.

Hooray.  Glad to see that Tim and Aaron and I are smoking the same crack
as the rest of you guys :-)

>> Separately, we need to decide where a recipe lives.  The current
>> thinking is
>> "https://launchpad.net/ubuntu/karmic/+source/some-package/+recipe/recipe-name";
>> which seems OK to me (we'd have to trust a bit that this recipe would
>> build a recipe for some-package in karmic, but that doesn't seem any
>> different to say branches today).
> 
> I don't think this is good for a few reasons:
>  * The URL is too long
>  * The traversal would fail for a recipe where a source package name
> doesn't exist yet
>  * The recipe is tied to a series.  Recipes should be independent of a
> series.
>  * Why have more than one recipe for one package?
> 
> My suggestion is:
> /<distro>/+recipe/packagename

Well... I'll reply to your followup about this I guess...

>> Finally, we could stick an archive on the recipe, but maybe we don't
>> want to.  I'll talk about this a bit more later in the mail.
> 
> We absolutely don't want to do this, because source packages can exist
> in many archives.  The act of publication of a package is separate from
> its existence.

Well, I knew the above was true, but this morning it seems much more
obvious to maintain this separation in the recipe too.

>> This leads to a schema a bit like:
>>
>> Recipe:
> 
> I think this should be called SourcePackageRecipe, mostly because we
> might have other recipes and somewhat for consistency reasons.

+1 to that.

>>  - id, registrant, date_created, owner, date_last_modified
>>    - all standard launchpad fields.  the owner would be able to edit
>>      the recipe.
>>  - name
>>    - the last bit of the url
> 
> I don't think you need this.
> 
>>  - distroseries, sourcepackagename
>>    - provides the rest of the url
> 
> I would say you need "distribution" instead of distroseries.

Well this is all tied in a pretty simple way to the "where does a recipe
live", so once we sort that out this is easy.

>>  - recipe
>>    - a text field containing the text of the recipe (probably with
>> mangled branch references so lp:foo would be replaced with lp:21435)
>>
>> RecipeBranch:
>>  - id, recipe, branch
>>    - all obvious i hope
>>
>> What follows hopefully doesn't depend too much on how the above gets
>> decided in the end.
>>
>> For the job of building a recipe into a source package we'll have a
>> BuildSourcePackageFromRecipeJob table.  I foresee this table looking like:
>>
>> BuildSourcePackageFromRecipeJob
>>  - job
>>  - recipe
>>  - archive?
> 
> I don't know if archive is going to be necessary here because it ties
> the creation of a source package to one archive.  We might want to take
> an existing recipe build job and re-upload it to a different archive.

Er.  Hm.  I guess I don't know enough to be certain about this, but I
think when we build a source package we're always going to want to then
build it for an archive?  I would guess that you can handle
republication by using the 'copy package' feature that exists already?

>> BuildQueue will get a row with a job column will reference same job and
>> have a particular job_type.
> 
> Yep.
> 
>> One of the things bzr-builder does when it creates the debianised source
>> tree is create a manifest, which is a sort of frozen version of a recipe
>> -- it references particular revisions of the branches so as to allow a
>> repeat of exactly this build.  We could use a manifest like this to
>> actually run the recipe: at the point where the build is requested, we
>> make the manifest and stuff it into the database.  This seems like a
>> neat idea, but isn't how bzr-builder works now as far as I can tell.
> 
> I think this manifest should be stored somewhere with the build job.  As
> I discussed with Jono on Monday, we know we're going to need a new table
> (SourcePackageRecipeBuild) that records a build event to get a source
> package from a recipe.  This table should have the manifest on it.
>
> It's also possible we don't need this table and we can just use the
> BuildSourcePackageFromRecipeJob.

I'm not sure what the difference would be, indeed.  What is the
conceptual difference between a Build and a BuildPackageJob?  I was
under the impression that there was still a split mostly to avoid
rewriting all of Soyuz -- if not, I'd love to know (and try to remember)
what it is.

>> This doesn't include anything that will actually create
>> BuildSourcePackageFromRecipeJob rows (say every day for a daily build
>> PPA).  I guess we can worry about this later.
>>
>> I think the current plan is to use bzr-builder to make the debianized
>> source tree and bzr-builddeb to then make the source package.  I'm
>> presume the process for getting the source package off the builder and
>> into the process of being built will follow that of the existing
>> builders: the builder will tell the buildd-manager where to get the
>> .dsc, the manager will parse this to find the other parts of the package
>> and then grab them, shove all of the files into the librarian and
>> trigger the existing parts of soyuz to look at them somehow[1].
> 
> What happens for binary builds is that the builders return a bunch of
> files that the buildd-manager throws into a directory on disk.  It then
> calls process-upload.py (using Popen :( ) to deal with it.  For a source
> package resulting from a recipe build we can do exactly the same thing.

Cool. 'exactly the same thing' even goes as far as handing them to the
process-upload.py script?

> One thing I need to change though is to stop this use of Popen since it
> blocks everything else on the buildd-manager.  There's a spec for this
> at
> https://blueprints.edge.launchpad.net/soyuz/+spec/buildd-manager-upload-decoupling

If I read the above right, this isn't actually strictly speaking
required to have build from recipe working?

I can certainly see how it would be a good idea though.

>> Something that's missing from all the above is how the archive is
>> selected.  It's more or less essential that the
>> BuildSourcePackageFromRecipeJob knows the archive, so that the generated
>> source package can be built for the right one.
> 
> Or we can divorce the archive from the Job entirely and have another
> mechanism that records who requested the build/upload.  This is
> important because we need to observe upload ACLs.

I still don't see why you'd want it separate from the job (job rows
persist beyond the execution of the .  It would be good to record
requester on the job though, for sure.

>   It could be tied to the
>> recipe or it could be supplied when the job row is created.  In some
>> sense the archive is totally orthogonal to the recipe, but OTOH, I can't
>> really see the use case for targeting more than one archive with a
>> recipe.  Advice welcome.
> 
> The recipe, no, the actual source package, yes.

I've got this message now :)

>> In case the above wasn't enough, here's some things I haven't thought
>> hard about:
>>
>>  - do people want to subscribe to a recipe?
>>    - does this mean getting notified when the recipe builds or fails to
>>      build?
>>    - does this mean getting notified when the recipe is changed?
> 
> If a recipe fails to build we need to notify the recipe owner and if the
> person requesting the build is different, them also.
> 
> Soyuz already has a lot of code for dealing with notifications, it
> should be easy enough to hook some more bits in.

OK.

>>  - the whole privacy thing.
>>    - do we only allow recipes to be created that reference branches the
>>      owner can see?
> 
> Makes sense.
> 
>>    - is having the people who can view the recipe being the intersection
>>      of those that can see the branches reasonable?
> 
> Yes.

Cool.

>>    - the issues of accessing private branches from the buildslaves
>>      scares me a bit, I hope we can avoid worrying about that until some
>>      time in 2010.
> 
> Yeah, I had only considered the firewall rules from the slaves.
> Presumably we'll need a buildd-slave SSH key that can access everything?

This sounds too binary for me: I don't think we want anyone who can
build a private recipe at all to be able to access all private branches.
 I think we can do something along the lines of embedding credentials in
the HTTP url.

>>> The model code should implement the interface ISoyuzJob (although this is a 
>>> terrible name, it will be changed) which is declared in 
>>> lib/lp/soyuz/interfaces/soyuzjob.py.
>> This file doesn't seem to exist?
> 
> See Muharem's response.
> 
>> In coarse outline, building a source package from a recipe isn't very
>> different from building a binary from a source package, so this sounds
>> like it will be a mass of (presumably devilish) details rather than deep
>> design work.
> 
> Hopefully yes.  One thing that we need to make sure of is that *all*
> build jobs must have a determinate build time.
> 
> We currently have a system in place that estimates how long it will be
> before your package build starts.  It does this by adding up the
> previous build times for all the packages in the queue in front of you.
> 
> To be able to continue to do this, we need to be able to look up a
> SourcePackageRecipeBuild for a package name (or the equivalent Job table
> row) and see how long it took to build last time.

I think we want to key off recipe, not source package here.  But it
sounds easy enough (select job.date_finished - job.date_started from
buildsourcepackagejob, job where buildsourcepackagejob.job = job.id and
job.status = completed and buildsourcepackagejob.recipe = $RECIPE order
by job.date_started desc limit 1 or similar).

> Muharem is refactoring our code right now and making an interface that
> your job classes must implement for us to be able to do this.
> 
>> [1] I guess the fact that these packages aren't signed will bite us in
>>    the ass somehow or other at some point, but I don't know how much it
>>    affects how this bit would work.  We don't *have* to get the source
>>    package files into Soyuz via the Librarian I guess.
> 
> Mark S has suggested that we have a single Launchpad key to sign them,
> so that if the packages are used outside of Launchpad then people know
> where they came from.

I think that probably makes sense.  Don't know where to do the signing
though -- maybe the buildd-master could do it?

Cheers,
mwh