launchpad-dev team mailing list archive

[Celso Providelo <celso.providelo@xxxxxxxxxxxxx>]

On Fri, Aug 7, 2009 at 1:56 PM, Julian
Edwards<julian.edwards@xxxxxxxxxxxxx> wrote:
> On Friday 07 August 2009 17:28:19 Jonathan Lange wrote:
>> On Fri, Aug 7, 2009 at 5:25 PM, Jonathan Lange<jml@xxxxxxxxxxxxx> wrote:
>> > Hello everyone,
>> >
>> > Another spec from me today, this one based on some notes that cprov,
>> > mwhudson, james_w and I made at UDS in Barcelona.
>> >
>> > This spec shows one way for Launchpad to build branches (esp source
>> > package branches) to archives, including PPAs and primary archives.
>>
>> *ahem*
>>
>> https://dev.launchpad.net/BuildBranchToArchive
>
> Thanks for starting that page off Jono.

Yes, thanks for leading this change. It will be an amazing feature
when it's out.

> Rather than writing a new "dispatcher" I think it might be a good idea to
> amend our existing buildd-manager to do these branch jobs, so that we can use
> the existing build farm.  I'm vastly simplifying here, but all we need to do
> is make a "branch job" chroot for that type of job.

I think installing `bzr-buildddeb` on the existing chroots is better,
mainly because it avoids maintaining 2 sets of chroots for all
supported distroarchseries.

The only inconvenient is performance, but until we identify it as a
major problem, I would simply go with the existing chroots and install
the needed stuff as part of the build process.

It will be fine, as long as `bzr-builddeb` is still functional in all
the supported suites (even if we have to include a parallel PPA
providing it).

1. Download the appropriate chroot
2. Unpack and set it up
3. Override sources_list
4. Upgrade
5. Install `bzr-builddeb`
6. Run it!
...
N. Clean up

Steps 1 - 4 and N already exist and can be reused from the debian
slave. Step 5 is trivial and step 6 should be something like

`bzr get -r REV BRANCH_URL job` then `bzr bd job -- -S -uc -us`

obs: James, I couldn't make bd build a remote branch directly in
jaunty, I had to fetch the branch first then build it. Am I doing
something wrong ?

> We currently have unsigned uploads when the buildd-manager uploads binaries
> and this could be extended to sources coming from the branch jobs.  The only
> issue I can think of is the lack of a dsc file signature.  Even if we did sign
> it with an LP signature, I'm not sure how meaningful that would be since we
> trust our build farm anyway.  It seems prudent to audit our code to see how
> much of it makes assumptions around dsc signing (I can think of a few).

Only one template assumes signed source uploads for PPAs, IIRC, all
other SPR.dscsigningkey callsites respect the schema (this column is
nullable).

The soyuz upload processor is already prepared accept unsigned source
uploads, we may need to encapsulate this as a different upload policy,
but this is will be a tiny change.

I'm little concerned about the schema changes we have to make. I'm
proposing them based on the fact that we agreed on using the existing
build-farm to assembly source packages from branches. It implies in:

1. Perform branch-jobs using the existing `launchpad-buildd`s.

Currently the only registered slave is the 'debian' one
(DebianBuildManager) we can easily create a new one, let's say,
BzrBuildManager reusing the chroot setup/cleanup and adding the
bzr-builddeb steps replacing sbuild.

The instead of calling build(..., 'debian', ...) over XMLRPC, we would
pass 'bzr' instead.

2. Dispatch and collect branch-jobs using `buildd-manager`

One way of implementing this would be extending BuildQueue to support
both, source and branch jobs.

The existing names are confusing, but in the same way the Build
content class represents which source has to be build, the proposed
BranchJob represents which branch has to be build.

BuildQueue represents a queue of jobs that have to be dispatched to
builders (better named 'BuilderQueue', anyway ...)

If we add a BranchJob FK on BuildQueue we can continue to use the same
dispatching mechanisms against the same pool of builders and also
manage priority in the same way for both types of build.

This way dispatching and collecting results can be delegated from BQ
to one of its children, Build or BranchJob, and implemented according
each context.

What do you guys think  about this ?

[]
-- 
Celso Providelo <celso.providelo@xxxxxxxxxxxxx>
IRC: cprov,  Jabber: cprov@xxxxxxxxxx, Skype: cprovidelo
1024D/681B6469 C858 2652 1A6E F6A6 037B  B3F7 9FF2 583E 681B 6469