yellow team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Hi Robert.  I'm off on national holiday but writing to try and get the
ball rolling on something that is a bit urgent for us to resolve.

One of the goals you gave us for lpsetup was for lpsetup to not only be
able to set up a Launchpad developer's system, but maintain it.
Specifically, AIUI, lpsetup should be able to uninstall workarounds (and
packages?) as they are no longer needed, and install and change new
files that are needed.

We've talked a bit about how to do this.  One approach would be to do it
in our own software.  I outline one possible way to do it in an email
below, but for the primary purpose of this email, you don't need to read
it: consider this to be an option of "we write Python code to do this
ourselves."  I think that initially this will be the easier approach,
and it's a relatively small incremental change from what we do now.

Alternatively, when Benji heard of the requirements, he perceptively
recognized that this sounds a lot like the requirements of Debian
packaging.  It can upgrade, downgrade, change files, add files, remove
files, and so on.  We could probably do what we need with two packages.
 One would be for changes in the LXC host.  The other would be for
changes in the LXC container that actually ran Launchpad.  lpsetup would
simply make sure those packages were installed properly (and use any
commands they might provide).  If people were to eschew lpsetup, they
could get most of the benefits simply by installing the appropriate
packages.

I think the Debian package approach might be better for the long run,
simply because uninstalling and installing system changes is not an easy
problem and Debian packaging's job is to solve it.  Using Debian
packaging would require some significant changes from what we are doing
now and would take us longer, but we think it might be the right thing
to do in the long haul.

You might be able to have a fully-formed opinion from the information
above.  However, for a bit more data, I wanted to list the changes that
we need.  I think most things would work well as Debian packages except
for two items ("One way or another, make sure user has ssh keys, bzr
whoami, and bzr lp-login" and "Create an lxc to work with").  I discuss
both below.

= Base (for all environments) =

 * Make sure ssh and bzr are installed
 * One way or another, make sure user has ssh keys, bzr whoami, and bzr
lp-login (we've considered adding a separate command, "inituser," for
this or simply giving people directions on what to do.  It is not well
suited to Debian packaging IMO, though could be done.  I prefer the
"tell users what to do" approach, as part of the instructions for using
lpsetup, though that may go counter to your larger vision.)

= LXC host =

 * install lxc, haveged
 * Workaround for 925024
   * ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/
   * apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
 * set up /etc/lxc/lp-setup.conf (this may no longer be necessary, but
we do it now)
 * Create an lxc to work with (Requires sudo.  This is user specific and
not best suited for Debian packaging IMO, though could be done.  As a
user, I personally dislike packages that require configuration for them
to be installed, even though I know it can be done. This could instead
be a command that the Debian package installs?)
   * install workaround for bug 974584
     * modify /var/lib/lxc/[CONTAINER]/fstab to append 'none dev/shm
tmpfs defaults 0 0\n'

= LXC container running a Launchpad system =

 * (install lpsetup)
 * install workaround for bug 959352
   * mkdir -p /rootfs/usr/lib
   * ln -s /usr/lib/graphviz /rootfs/usr/lib/graphviz
 * install workaround for bug 1014916 (Lucid container only)
   * Change /var/lib/lxc/lptests/rootfs/etc/init/networking.conf to not
wait for udev
   * remove /etc/init/udevtrigger.conf

= Launchpad system (irrespective of LXC) =

 * make sure apt repositories are available
    'deb http://archive.ubuntu.com/ubuntu {distro} multiverse',
    'deb http://archive.ubuntu.com/ubuntu {distro}-updates multiverse',
    'deb http://archive.ubuntu.com/ubuntu {distro}-security multiverse',
    'ppa:launchpad/ppa',
    'ppa:bzr/ppa',
 * apt-get update
 * install apt packages (I realize you don't want all of these
eventually, but I don't think the separation can be part of this effort)
    'launchpad-database-dependencies-9.1',
    'launchpad-developer-dependencies', 'apache2',
    'apache2-mpm-worker', 'libapache2-mod-wsgi'

Thank you for your thoughts.

Gary


-------- Original Message --------
Subject: lpsetup system maintenance thoughts from Brad and me
Date: Thu, 28 Jun 2012 17:34:29 -0400
From: Gary Poster <gary.poster@xxxxxxxxxxxxx>
To: Launchpad Yellow Squad <yellow@xxxxxxxxxxxxxxxxxxx>

We have a few cards that might be related.  Brad and I talked about them
today.

The cards are these:

 * make root command to update system to newest/best configuration
 * Add banners to all created and modified configuration files in the
system.
 * interactive developer story

The first one is the most mysterious, and the crux of the conversation.
 Robert wants lpsetup to be able to not just set up a Launchpad
developer system but maintain it.  In the life of a developer's system,
after the developer has successfully created an environment for
launchpad development [1] and used it for awhile [2], a new version of
lpsetup may be available.  It may have new configurations and
workarounds.  Robert would like the developer to be able to run a
lpsetup command that will be able to smoothly handle the system upgrades
from the previous developer configuration to the current one.

The initlxc and inithost commands must be more than idempotent for this
to work.  They must be able to uninstall previous, discarded or improved
configurations.

I proposed that this could be accomplished by subdividing each
configuration element done by inithost and initlxc.  Each one could have
four callables.

 * Am I needed?  This may need to check what packages, or versions, are
on the system; or the version of the OS (Lucid, Precise, etc.); or
whether or not the host is an LXC container.
 * Am I installed?  This should be able to determine whether the
configuration has been performed, making whatever checks are necessary
(looking at file content, looking for installed packages, or simply
looking for some marker in the system).
 * Install.  This does the work, whatever it is.  It should always do it
in a way that can be cleanly reversed.
 * Uninstall.  Reverse an installation.

Each configuration change we ever make would need to stay around forever
in our files.  Configuration changes would be ordered.  We would use the
structure something like this.

 * For inithost, we would walk through all the inithost command's
configurations.  For each, code would ask, are you needed?  Are you
installed?  If you are not needed and installed, please uninstall; if
you are needed and not installed, please install; otherwise, no-op.
 * For initlxc, we would walk through the initlxc command's
configurations, and then call inithost in the container.  (The inithost
code can reliably discover it is running in a container and do
lxc-related jobs there, by the way.  I learned how from Serge.)

That's it.

The plan is related to the other two cards because it could fit in
nicely with them.

Banners added to created or modified files could be part of the markers
that the configuration uses to determine whether the change is
installed.  They don't have to be, but they could.  (They could even
include diffs as a comment, which could be used as an uninstall story if
we though that was a good idea.)

The interactive developer story could be supported by adding one or two
more items to each configuration: a description or title of the change
and a rationale.  Then the developer story could include a dry run
showing everything that would be installed and uninstalled, and why; and
could be the UI meat of an interactive yes/no step-by-step installation
story.

OK, have to run.  I look forward to talking about this with you all. Thanks!

Gary



[1] Creating the environment is roughly equivalent, with a bit of
variable glue left out, to "lpsetup inithost" or "lpsetup initlxc";
"lpsetup get"; and "lpsetup update."

[2] Using the enviroment on a daily basis would primarily consist of
"lpsetup update" with maybe a sprinkle of "lpsetup get" now and then.