maas-devel team mailing list archive

[Gavin Panella <gavin.panella@xxxxxxxxxxxxx>]

Over the New Year holidays I worked on changing the way the components
of MAAS are packaged, in the Python sense (not Debian/Ubuntu packages).

The branch is lp:~allenap/maas/repackage, merge proposal at
https://code.launchpad.net/~allenap/maas/repackage/+merge/200935.

It's a massive branch, and took a lot of work to do it. There are
undoubtedly some rough edges, but overall I'm fairly sure it's going to
be a much better environment for developing MAAS.

All but 13 tests pass, and they're all in one particular subcomponent.
I need help to squash those bugs, and to get this branch in a landable
state.

The diff itself is so large that it's hardly possible to take it all in,
so I'm suggesting we do this with a different approach: using it.

You can *really* help by trying to use it. Check it out, build it, take
a look around, see what's missing, broken, better, and so on. Comment on
the merge proposal, talk to me, try to fix it.

For my part, I will regularly merge trunk and keep the branch
up-to-date, fixing conflicts. I'll do my best to keep up with the
comments coming in on the merge proposal, fixing things, replying, and
so forth.

At some point I'll pull the trigger and land it. I doubt that'll be
without drama, but at that point I'll need your help again to make
changes to the integration tests, the Ubuntu packaging, and whatever
crops up. I've tried hard to resist changing behaviour in the main body
of the codebase so it should be things like moved files and changed
names rather than functional bugs.

I don't want this to bit-rot because I'm not going to do it again.
Please help! :)

Some miscellaneous thoughts:

- The current setup.py is largely useless. There's enough garbage in
  there for buildout to work with, but that's about all its good for.

- I've split the codebase into six separate Python distribution
  packages:

    maas-apiclient
    maas-client
    maas-cluster
    maas-develop
    maas-region
    maas-testing

- maas-develop is a new package that is focused only on developing MAAS;
  it never need be installed on a machine.

- Each of these uses setuptools to provide console_scripts hooks, for
  example, and pkg_resources for finding (some) entry points and
  important directories instead of __file__-path hiking. I'm _hoping_
  that each of these should translate mechanically into a python-maas-*
  Ubuntu package.

- I've dropped use of buildout. Buildout is kind of neat, but it never
  manages to plug all the leaks in the abstractions it provides. When
  something goes wrong it can be an exhausting hunt to discover the
  problem. Given that virtualenv and `pip install --editable` provide a
  similar environment for development, its time was up.

- I have *not* used pip's support for requirements files. They're
  superfluous; if a package's dependencies are correctly defined in
  setup.py then there is no need for them.

- I have carefully gone through each and every Python dependency trying
  to ensure that nothing is missing. This involved running the tests
  suites in bare LXC containers, so I'm reasonably confident they're
  good.

- I have dropped use of nose. I've switched to using plain unittest and
  making more use of testresources. This is a bit of a regression in
  functionality if I'm honest, but it does mean that `python setup.py
  test` does something useful. I think we can reintroduce nose at some
  point as a development convenience, but I'd like to gate landing with
  stock unittest.

- The new setup.py files are all *generated*, so don't review them. See
  pkg/common/__init__.py and pkg/*/configure.py to understand what's
  happening. This does not make me happy, but it was the best compromise
  I could come up with at the time.