yellow team mailing list archive

[Francesco Banconi <francesco.banconi@xxxxxxxxxxxxx>]

Hi all,

lpsetup status quo and possible enhancements follow.
This is just an attempt to analyze the project so that it could be
easier to plan future development. I felt the need to write this report
after the call with Gary last Friday (about the LEP) and after reading
Robert's comments.
I'm sorry, it's `-vvv --engrish`.


= Current status =

The UI/entry-point of lpsetup is the lp-setup command. This command is
made of several sub commands, and each sub command can contain different
steps. When executing a sub command it is possible to specify the steps
to run or to skip.
This machinery is implemented using:

- a custom ArgumentParser (argparse.ArgumentParser):
  - with the ability to register and handle sub commands
  - with the ability to re-create the cmd line args from a namespace
    object (this is useful when the script run `sudo` on itself)

- a SubCommand object:
  - supports namespace initialization and validation
  - implements the *restart myself as root* functionality
  - contains sub command help
  - adds to the parser the required options
  - optionally implements the steps system

Sub commands and steps follow.


== lp-setup install ==
Installs a Launchpad instance locally

Steps:

- initialize:
  - apt install bzr and ssh
  - create the user if it does not exist and the command is invoked by
    root
  - create the ssh keys for the user, if not present
  - set up known hosts and authorized keys properly
  - bzr init-repo
  - bzr branch (or pull) launchpad
  - set up dependencies (eggs, yui, source code, download-cache)
  - haveged tweak

- setup_bzr_locations:
  - create .bazaar/locations.conf (submit_branch, public_branch, etc...)

- setup_apt:
  - add ppas
  - apt install launchpad dependencies

- setup_launchpad:
  - add the user to group sudo
  - update-sourcecode
  - link-external-sourcecode
  - create apache document roots
  - enable apache modules
  - launchpad-database-setup
  - make && make install
  - change the owner of /srv/launchpad.dev/
  - add launchpad addresses to /etc/hosts


== lp-setup lxc-install ==
Create a Launchpad environment inside an LXC

Steps:

- install.initialize: see above

- install.setup_bzr_locations: see above

- create_scripts (optional):
  - generate the three scripts used by buildbot

- create_lxc:
  - apt install lxc and libvirt-bin
  - install subunit if requested
  - apparmor tweak for lxc-start
  - gateway check (lxcbr0 or virbr0)
  - lxc-create
  - set up authorized keys for non-interactive login into the container
  - lxc fstab tweak (to be able to create semaphores in lxc)

- start_lxc

- wait-for-lxc

- initialize_lxc:
  - apt install bzr and ssh in lxc
  - add ppas in lxc
  - apt install lpsetup inside lxc
  - graphviz tweak

- setup_launchpad_lxc:
  - call lp-setup install inside the container executing the following
    steps:
    - install.setup_apt: see above
    - install.setup_launchpad: see above

- stop_lxc


== lp-setup branch ==
Create and build a new Launchpad branch

Steps:

- create_branch:
  - bzr branch devel target

- setup_and_make:
  - link-external-sourcecode on the new branch
  - make
  - make schema if requested


== lp-setup update ==

Steps:

- update_launchpad:
  - apt update and upgrade
  - bzr pull devel
  - update-sourcecode
  - link-external-sourcecode
  - make (if a new revision exists)
  - optionally make schema (if a new revision exists)

- link_sourcecode_in_branches:
  - link-external-sourcecode in all the branches found in the shared
    repository


= TODO =

- make an interactive developer story, including warning and configuration

- investigate why LANG=C does not work in a local system


= Possible enhancements =

Ideas, food for thought follow.

== Having separate commands for separate tasks. ==

It should be quite easy to create something like
`command_from_subcommand(SubCommand)`, a function that creates an entry
point so that a subcommand can be invoked as a script. This way we can
have a central hub (lp-setup) and several other executables (lp-install,
lp-update...), maybe created by the distribution scripts (setup.py).

== Reorganize the code ==

Robert, if I am not wrong, suggested these separate actions:

- initialize a container (lxc, ec2, local, lxc-ephemeral)
- get the sourcecode
- set up database, make, make install, make schema, etc.

This require more work.
As explained above, the current organization of of sub commands and
steps does not reflect this approach. We should reorganize the code and
reorder the steps.
Trying to reuse as much of the current code as possible, we could create
a system like the following:

- lp-setup init [container]
  - e.g.: lp-setup init lxc
  - accepts a container argument: we could create a container object as
an interface for really different containers to be created, started,
stopped. The contract could also let the user connect and execute
commands inside the container. The container is initialized installing
all the launchpad dependencies and the required tweaks, but no source
code is retrieved

- lp-setup get (or retrieve, or something)
  - branch the launchpad repository, read below a description of
possible set up methods.

- lp-setup build
  - build launchpad (including download-cache, external dependencies,
install, schema)

Using this approach, the current lp-setup install and lxc-install sub
commands can be converted as follow:
- lp-setup install [container]
  - calls init && get && build
  - supports the testing story


== Checkout ==

Robert suggested lightweight checkouts, and this can be the default.
We can support normal shared repository/branches via a --with-tree option.
It's easy for the user to convert a normal branch to a co-located one.


== lpsetup as part of lp-dev-utils ==

lp-dev-utils is already packaged, it should be not so hard to include
lpsetup.
However, I am inclined to continue the development of lpsetup as a
standalone project at least until it will be ready for a initial
release. It's a bit disappointing that this release will be delayed if
we decide to proceed with (all or part of) the changes described here.
By the way, those changes look good IMHO.
However, we are fine with lpsetup inside lp-dev-utils if the latter can
depend on python-shelltoolbox.


Thoughts?
Thank you.

-- 
Francesco Banconi