launchpad-dev team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxx>]

The Ubuntu phone images are due to hit the release-to-manufacturing
(RTM) point later this cycle.  With the pace of the phone work, it
doesn't look practical to deal with this by SRUing all the required
changes into trusty - we're talking about substantial feature
development which probably wouldn't be manageable in trusty even with a
rather liberal SRU policy for phone-specific components.

I've therefore been thinking about how we might manage to do a stable
phone image somewhere in the middle of a development release cycle;
while the extensive work we've done on rolling quality helps a lot, it
probably won't quite be enough on its own, and we don't want to have the
phone work interfere too much with other Ubuntu developers, so we need
some reasonable way to create a stable branch for RTM, preferably
without just hiding everything off in some non-Launchpad repository that
would require lots of work in our CI tools to handle.  I spent some time
in Malta this week discussing this with William Grant of Launchpad and
various phone managers and developers, and I think we now have a
solution that meets the requirements, is technically sound, and might
potentially be useful elsewhere in the future.

= PPA strategy (shelved) =

The first idea I had was to copy everything relevant to the phone image
(probably a full build-dependency closure) into a PPA; create a second
PPA which could act as an equivalent of -proposed; adjust Launchpad and
launchpad-buildd, including the new live filesystem building code, to
allow this PPA to be entirely self-contained, so that builds don't refer
to the primary archive; and run whatever syncing and merging tools we
need to propagate changes from the primary archive to the RTM PPAs as
required.

This very nearly works.  There are some publisher performance problems
with putting a very large number of packages in a single PPA, but
William would have been happy to sort those out.  However, a more
serious problem is that chroots are shared between utopic in the primary
archive and utopic PPAs, and there's currently no way to disconnect
those, so over time it would become more difficult to build updated
packages for this PPA.  While we might be able to fix this, and doing so
would be a workable plan B, it points to PPAs being the wrong model.

= Derived distribution strategy (candidate) =

A feature called "derived distributions" was added to Launchpad a few
years ago, originally for use by Linaro: it allows branching an entire
distribution or part of it into a new distribution, with some tools to
help keep them in sync where appropriate.  It has never actually been
used properly in production; it was tried once, ran into some small but
fatal problems, but staffing issues at the time meant that these were
never fixed.  William has given it a fresh try on a non-production
Launchpad instance and it doesn't seem to be too far off working
reasonably, so we think that we can salvage this work.

The usual Launchpad copying tools will still work much as before, and
we'll be able to have PPAs based on the new derived distribution.


The current plan is therefore:

 * CI and Foundations team members will spend the next two months making
   sure that all the tools we need are in place.  This includes fixing
   up derived distributions, preparing an initialisation script, making
   sure that the CI engine (train or airline, as appropriate) can handle
   building for a derived distribution, and adding support for
   ubuntu-touch image builds from derived distributions.

 * Monitor progress of phone images closely in the early part of August.
   We should have a beta image around this point.

 * Rick has asked me to take ownership of deciding when to branch for
   RTM; note that this decision can be taken retroactively by copying
   older versions of packages, so we can say "the image from three days
   ago looked good but things have been broken since then, so let's
   branch from that".  I intend to take this decision based on how
   stable images are and how much tension there is between phone work
   and development of other parts of Ubuntu; all other things being
   equal I would prefer to keep the branch shorter, but we will need at
   least a couple of weeks' clearance before RTM in order that we aren't
   debugging our infrastructure right up to the wire.  Since I have
   pre-booked holiday in mid-August, William Grant will be my backup for
   this.

 * As with the shelved PPA plan, we'll only copy packages that are
   needed to build the phone image.  This makes it easier to track
   things and keeps our use of resources down, which has benefits such
   as faster publishing.

 * We will then have an "ubuntu-phone" (name TBD) distribution in
   Launchpad with a series named something like "rtm-14.09", and will
   land changes for RTM there, allowing other parts of Ubuntu to move on
   without risk of breaking phone images.  We'll have a -proposed pocket
   as usual and proposed-migration will operate on it.  I'd like
   permissions on this distribution to be identical to those in Ubuntu
   where possible, but we can discuss details there.  There'll be
   system-image channels for RTM, probably per-device.

 * We don't want to have to maintain RTM series for a long time.  The
   plan is to move devices to the next stable branch along as soon as
   possible.  To facilitate this, the usual rule should be that changes
   should only go into this distribution after they're already on their
   way into mainline Ubuntu (as for SRUs), to make sure that we don't
   end up in a situation where developers land changes on the stable
   branch but don't quite get round to landing them in mainline.


= FAQ =

Can we use this strategy for other Ubuntu flavours or derivatives?

  In theory, maybe.  It's very much easier to use this strategy for
  phone images, though, because the standard distribution mechanism is
  via system-image channels so we don't need to worry about updating
  sources.list or arranging for mirroring of the derived archive, etc.
  There's also a potentially serious social cost to having lots of
  archive branches, and I wouldn't want to do so except for short
  durations under tight management; I don't want to see the
  Ubuntu-and-flavours development community fracturing unnecessarily.

  That said, the raw technology may prove to be useful in other similar
  contexts in the future.  Ask us after we've had some experience with
  running it for RTM ...

What if derived distributions can't be made to work after all?

  We fall back to PPAs and keep a really close eye on everything.
  Fortunately this is an unlikely scenario, I think, as this will
  probably end up being more work.

How will we manage syncing and merging?

  The nature of the RTM branch is such that we probably won't need very
  much in the way of keeping up to date with Ubuntu.  Most changes will
  land via the CI engine.  That said, there will doubtless be some
  things like security updates to core packages that we need to land, so
  time permitting I'd like to set up something like merge-o-matic to
  keep track of this.  If possible, these will be copied via the CI
  engine to minimise the risk of unexpected image testing failures.

Will non-Canonical developers be able to take part in this?

  The current engine ("CI Train") is visible to non-Canonical
  developers, but the use of a spreadsheet, a private Jenkins instance,
  and semi-manual organisation of landings makes it difficult to use in
  practice.  The new engine (sometimes called "CI Airline") is due to
  land this month and should be much easier to use without
  company-specific privileges.  I see no reason why it can't all be
  open.

  We'll be trying hard to have a common rootfs across customers, but
  it's possible that deadlines will force there to be a couple of
  patches which land separately for one or the other.  Where possible,
  though, this kind of thing will be in the device or custom tarball
  rather than the main rootfs.


Any other questions?

-- 
Colin Watson                                       [cjwatson@xxxxxxxxxx]