launchpad-dev team mailing list archive

[Scott Kitterman <ubuntu@xxxxxxxxxxxxx>]

On Wednesday, June 04, 2014 21:20:50 Colin Watson wrote:
> The Ubuntu phone images are due to hit the release-to-manufacturing
> (RTM) point later this cycle.  With the pace of the phone work, it
> doesn't look practical to deal with this by SRUing all the required
> changes into trusty - we're talking about substantial feature
> development which probably wouldn't be manageable in trusty even with a
> rather liberal SRU policy for phone-specific components.
> 
> I've therefore been thinking about how we might manage to do a stable
> phone image somewhere in the middle of a development release cycle;
> while the extensive work we've done on rolling quality helps a lot, it
> probably won't quite be enough on its own, and we don't want to have the
> phone work interfere too much with other Ubuntu developers, so we need
> some reasonable way to create a stable branch for RTM, preferably
> without just hiding everything off in some non-Launchpad repository that
> would require lots of work in our CI tools to handle.  I spent some time
> in Malta this week discussing this with William Grant of Launchpad and
> various phone managers and developers, and I think we now have a
> solution that meets the requirements, is technically sound, and might
> potentially be useful elsewhere in the future.
> 
> = PPA strategy (shelved) =
> 
> The first idea I had was to copy everything relevant to the phone image
> (probably a full build-dependency closure) into a PPA; create a second
> PPA which could act as an equivalent of -proposed; adjust Launchpad and
> launchpad-buildd, including the new live filesystem building code, to
> allow this PPA to be entirely self-contained, so that builds don't refer
> to the primary archive; and run whatever syncing and merging tools we
> need to propagate changes from the primary archive to the RTM PPAs as
> required.
> 
> This very nearly works.  There are some publisher performance problems
> with putting a very large number of packages in a single PPA, but
> William would have been happy to sort those out.  However, a more
> serious problem is that chroots are shared between utopic in the primary
> archive and utopic PPAs, and there's currently no way to disconnect
> those, so over time it would become more difficult to build updated
> packages for this PPA.  While we might be able to fix this, and doing so
> would be a workable plan B, it points to PPAs being the wrong model.
> 
> = Derived distribution strategy (candidate) =
> 
> A feature called "derived distributions" was added to Launchpad a few
> years ago, originally for use by Linaro: it allows branching an entire
> distribution or part of it into a new distribution, with some tools to
> help keep them in sync where appropriate.  It has never actually been
> used properly in production; it was tried once, ran into some small but
> fatal problems, but staffing issues at the time meant that these were
> never fixed.  William has given it a fresh try on a non-production
> Launchpad instance and it doesn't seem to be too far off working
> reasonably, so we think that we can salvage this work.
> 
> The usual Launchpad copying tools will still work much as before, and
> we'll be able to have PPAs based on the new derived distribution.
> 
> 
> The current plan is therefore:
> 
>  * CI and Foundations team members will spend the next two months making
>    sure that all the tools we need are in place.  This includes fixing
>    up derived distributions, preparing an initialisation script, making
>    sure that the CI engine (train or airline, as appropriate) can handle
>    building for a derived distribution, and adding support for
>    ubuntu-touch image builds from derived distributions.
> 
>  * Monitor progress of phone images closely in the early part of August.
>    We should have a beta image around this point.
> 
>  * Rick has asked me to take ownership of deciding when to branch for
>    RTM; note that this decision can be taken retroactively by copying
>    older versions of packages, so we can say "the image from three days
>    ago looked good but things have been broken since then, so let's
>    branch from that".  I intend to take this decision based on how
>    stable images are and how much tension there is between phone work
>    and development of other parts of Ubuntu; all other things being
>    equal I would prefer to keep the branch shorter, but we will need at
>    least a couple of weeks' clearance before RTM in order that we aren't
>    debugging our infrastructure right up to the wire.  Since I have
>    pre-booked holiday in mid-August, William Grant will be my backup for
>    this.
> 
>  * As with the shelved PPA plan, we'll only copy packages that are
>    needed to build the phone image.  This makes it easier to track
>    things and keeps our use of resources down, which has benefits such
>    as faster publishing.
> 
>  * We will then have an "ubuntu-phone" (name TBD) distribution in
>    Launchpad with a series named something like "rtm-14.09", and will
>    land changes for RTM there, allowing other parts of Ubuntu to move on
>    without risk of breaking phone images.  We'll have a -proposed pocket
>    as usual and proposed-migration will operate on it.  I'd like
>    permissions on this distribution to be identical to those in Ubuntu
>    where possible, but we can discuss details there.  There'll be
>    system-image channels for RTM, probably per-device.
> 
>  * We don't want to have to maintain RTM series for a long time.  The
>    plan is to move devices to the next stable branch along as soon as
>    possible.  To facilitate this, the usual rule should be that changes
>    should only go into this distribution after they're already on their
>    way into mainline Ubuntu (as for SRUs), to make sure that we don't
>    end up in a situation where developers land changes on the stable
>    branch but don't quite get round to landing them in mainline.
> 
> 
> = FAQ =
> 
> Can we use this strategy for other Ubuntu flavours or derivatives?
> 
>   In theory, maybe.  It's very much easier to use this strategy for
>   phone images, though, because the standard distribution mechanism is
>   via system-image channels so we don't need to worry about updating
>   sources.list or arranging for mirroring of the derived archive, etc.
>   There's also a potentially serious social cost to having lots of
>   archive branches, and I wouldn't want to do so except for short
>   durations under tight management; I don't want to see the
>   Ubuntu-and-flavours development community fracturing unnecessarily.
> 
>   That said, the raw technology may prove to be useful in other similar
>   contexts in the future.  Ask us after we've had some experience with
>   running it for RTM ...
> 
> What if derived distributions can't be made to work after all?
> 
>   We fall back to PPAs and keep a really close eye on everything.
>   Fortunately this is an unlikely scenario, I think, as this will
>   probably end up being more work.
> 
> How will we manage syncing and merging?
> 
>   The nature of the RTM branch is such that we probably won't need very
>   much in the way of keeping up to date with Ubuntu.  Most changes will
>   land via the CI engine.  That said, there will doubtless be some
>   things like security updates to core packages that we need to land, so
>   time permitting I'd like to set up something like merge-o-matic to
>   keep track of this.  If possible, these will be copied via the CI
>   engine to minimise the risk of unexpected image testing failures.
> 
> Will non-Canonical developers be able to take part in this?
> 
>   The current engine ("CI Train") is visible to non-Canonical
>   developers, but the use of a spreadsheet, a private Jenkins instance,
>   and semi-manual organisation of landings makes it difficult to use in
>   practice.  The new engine (sometimes called "CI Airline") is due to
>   land this month and should be much easier to use without
>   company-specific privileges.  I see no reason why it can't all be
>   open.
> 
>   We'll be trying hard to have a common rootfs across customers, but
>   it's possible that deadlines will force there to be a couple of
>   patches which land separately for one or the other.  Where possible,
>   though, this kind of thing will be in the device or custom tarball
>   rather than the main rootfs.
> 
> 
> Any other questions?

How will uploads to this new derived distribution get back into Ubuntu?

Will uploads to this new derived distribution be limited to Ubuntu developers?  
If not, how is this being controlled?

Scott K