launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On 19 November 2015 at 04:03, Colin Watson <cjwatson@xxxxxxxxxxxxx> wrote:
> This is following up on a discussion with Kit Randel on our team call
> last night, and one with Robert Collins on IRC; and, more generally, the
> massively inconsistent mess that many of our projects are in.
...
> Here are a couple of relevant pages:
>
>   http://docs.openstack.org/developer/pbr/#requirements
>   https://www.caremad.io/2013/07/setup-vs-requirement/

pbr's handling of requirements.txt is something we're going to
deprecate *once* we have a replacement - and it will be long and
graceful. Basically requirements.txt should be as Donald says - known
good for a deployment, not the known-bad exclusions you want in a
library.

But it is still reasonable and beneficial to use pbr with
requirements.txt having the same content you'd put in setup.py: the
cost is confusion (you have a requirements.txt that doesn't contain
what a best-practice requirements.txt would contain). The benefits
are:
 - declarative, machine readable dependencies
 - including extras support (And we're working to do transparent
backwards compat with old pips, so users don't need to do that glue
themselves)
 - versions from git automatically and so on - all the pbr things

> It's useful to be able to provide a requirements.txt which just builds
> whatever's current; this makes things easier for developers, services
> such as readthedocs, etc.  I certainly agree that we shouldn't have our
> direct dependencies declared twice, and there are a couple of possible
> approaches to that.  Aesthetically I think I prefer declaring them in
> setup.py and using "-e ." in requirements.txt, since we generally seem
> to need editable mode anyway; but the pbr approach would work too, so I
> don't feel very strongly about this.

I don't understand what this actually means. What do you mean by
'current' - 'all from VCS' or something else?

> I propose the following as a policy for all Launchpad-related projects:
>
>  * All projects must declare their direct dependencies (only - must not
>    be flattened/resolved) in setup.py and use "-e ." in
>    requirements.txt, to be used with "pip install -r requirements.txt".
>    (Alternative: declare them in requirements.txt and use pbr's
>    requirements parsing support.)

This is the thing I'm confused about :)

Concretely do you mean:
setup(..., install_requires=['foo'])

+
requirements.txt contains
-e VCS-path-to-foo

 ?

>  * Projects that are the top level of a deployment must include a
>    flattened list of dependency versions in constraints.txt, as produced
>    by "pip freeze", to be used with "pip install -c constraints.txt -e
>    .".  All the necessary packages should be included in a suitable
>    dependencies branch.  Unless they are deploying to a system that
>    already has pip 7.1 or newer, they will typically need a
>    bootstrap-requirements.txt that upgrades to modern versions of pip
>    and setuptools in order for constraints to be available.

+1 :). [This is exactly what I wrote constraints support for]

>  * Any project that requires a fork of a dependency (i.e. a version not
>    on PyPI) must pin the forked version in constraints.txt.  If the
>    forked version is required for development builds, then it should
>    also include a suitable VCS entry in requirements.txt (see
>    https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support).
>    Note that this is safe because deployments use constraints.txt, not
>    requirements.txt.
>
>  * Testing libraries against application deployment versions should be
>    achieved using "pip install -c /path/to/application/constraints.txt
>    -e .".  Unless they are themselves the top level of a deployment,
>    libraries should not include full pinned constraints; they will only
>    get out of date.
>
> Does this all sound sensible, or have I missed something?

With the one caveat the bit I'm confused about, sure. I'd really like
to see more folk move away from procedural dependencies though, so am
going to try to dig into that a bit, if you have time.

-Rob


-- 
Robert Collins <rbtcollins@xxxxxx>
Distinguished Technologist
HP Converged Cloud