launchpad-dev team mailing list archive

[Barry Warsaw <barry@xxxxxxxxxx>]

On Nov 18, 2015, at 03:03 PM, Colin Watson wrote:

> * projects using buildout (Launchpad itself; lazr.* and friends)

Note that several of the lazr packages that I've touched recently-ish
(e.g. smtptest[*], config, delegates) I've already de-buildout-ified and
converted to tox.  tox is not only a great multi-version test regime, but the
virtualenvs it builds are perfectly suitable for all the normal venv-y stuff
you might want to do while developing a library.  You can also relegate the
test-requirements (i.e. non-runtime reqs) to the tox.ini so as to keep the
setup.py fairly clean for better integration with distro tools like pybuild.

I recommend we standardize on tox.

>   Tried and tested.  OTOH it's also a massive pain.  We're stuck on an
>   old version at least for Launchpad itself because we're relying on
>   buildout's partial isolation support which was removed in buildout 2;
>   and buildout doesn't support setup_requires, which many of our
>   dependencies now make use of directly or transitively, particularly
>   via pbr.  It is massively unclear how to run tests at all on many
>   lazr.* packages, never mind how to do so if you happen to be
>   attempting a Python 3 port; figuring out the exact details of how to
>   run ./bootstrap.py etc. is awkward at best and we have quite a lot of
>   bitrot here.

Yes, let's get rid of buildout.

> * projects using virtualenv and pip (turnip, txpkgupload, rutabaga)
>
>   Relatively new to us, but is mostly working well.  We have some
>   problems with coherent requirements management, and different
>   projects use requirements vs. constraints differently, which I'd like
>   to fix before we extend this to more things.

+1, and see above.

>Aesthetically I think I prefer declaring them in setup.py and using "-e ." in
>requirements.txt, since we generally seem to need editable mode anyway; but
>the pbr approach would work too, so I don't feel very strongly about this.

I've only recently seen a few knock-on problems related to pbr, but I don't
remember the details (Robert probably does) and I think they've since been
fixed.  I actually didn't know about the `-e .` trick until I read Donald's
post... neat!

> * All projects must use a virtual environment and pip to install their
>   dependencies.  (This is relatively straightforward for most projects;
>   Launchpad itself will take a while.)

Via tox, where appropriate.  An added benefit is that you generally don't need
to manually setup and manage venvs.

> * All projects must declare their direct dependencies (only - must not be
>   flattened/resolved) in setup.py and use "-e ." in requirements.txt, to be
>   used with "pip install -r requirements.txt".  (Alternative: declare them
>   in requirements.txt and use pbr's requirements parsing support.)

+1 for setup.py and `-e .`.  I don't have enough experience with pbr to know
whether it brings enough to the table to offset an additional dependency.

> * Projects that are the top level of a deployment must include a
>   flattened list of dependency versions in constraints.txt, as produced
>   by "pip freeze", to be used with "pip install -c constraints.txt -e
>   .".  All the necessary packages should be included in a suitable
>   dependencies branch.  Unless they are deploying to a system that
>   already has pip 7.1 or newer, they will typically need a
>   bootstrap-requirements.txt that upgrades to modern versions of pip
>   and setuptools in order for constraints to be available.

I've been stalled getting pip upgraded in Debian and Ubuntu, and I *really*
need to get back to it.  My TODO list is here:

http://anonscm.debian.org/cgit/python-modules/packages/python-pip.git/tree/debian/changelog

There's a fair bit of work to do before actually touching pip, and any help
would be greatly appreciated!

> * Any project that requires a fork of a dependency (i.e. a version not
>   on PyPI) must pin the forked version in constraints.txt.  If the
>   forked version is required for development builds, then it should
>   also include a suitable VCS entry in requirements.txt (see
>   https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support).
>   Note that this is safe because deployments use constraints.txt, not
>   requirements.txt.
>
> * Testing libraries against application deployment versions should be
>   achieved using "pip install -c /path/to/application/constraints.txt
>   -e .".  Unless they are themselves the top level of a deployment,
>   libraries should not include full pinned constraints; they will only
>   get out of date.
>
>Does this all sound sensible, or have I missed something?

It all sounds very reasonable.  These are old code bases so modernizing them
is a good thing.

Cheers,
-Barry

P.S. Can we switch to git?  (/me hears the sound of worms wriggling in cans)

[*] For those of you wanting to do some fun stuff with asyncio in Python 3,
take a look at our next generation SMTP testing framework, aiosmtpd:

https://gitlab.com/python-smtpd-hackers/aiosmtpd