openstack team mailing list archive

[Monty Taylor <mordred@xxxxxxxxxxxx>]

On 07/02/2012 08:43 PM, Joshua Harlow wrote:
> Ack, please don’t keep on adding on to the copy around stuff scheme.
> Pleaaaase :-)

You know - I was a huge opponent to the copy around stuff scheme when it
started. It raised all of my hackles and I got all upset about things
not being done the "right" way.

Turns out that we have a complex project with some interesting
requirements, and sometimes having an automated way to inject filtered
sets of code from a common source isn't as terrible a thing as you might
think (especially considering how half-baked ALL of pythons library and
dependency management stuff is. oy - I'm amazed I have hair left...)

> Is the devstack dependency list complete, when I created the anvil one,
> I found more than one hole...

Hi! This has nothing to do with the devstack dependency list. This list
is two lists, actually - one is the the superset of all of the
pip-requires files in all of the projects, the other is the superset of
all of the test-requires files in all of the projects.

> Also the devstack one is in a micro-format, maybe we can move to say, YAML?

The purpose of this is to drive pip installations of packages for
virtualenvs. That already has a required format that we have to use.
(again, this has nothing to do with devstack)

> How about hosting these requires on some website (with versions there)?

The apprpriate contents still need to exist in each projects'
pip-requires and test-requires files for unit testing, so you'll still
have to copy code into the projects.

> Github already provides this for the anvil dependency list, maybe that
>  (or something) similar is sufficent?

It's not even close.

It's possible that I was not clear on the purpose of what this is trying
to accomplish.

The underlying problem is that each project declares a set of
python-level dependencies. This is important so that the base unit of
release, the source tarball, is itself both complete, and also testable,
since we have to be able to have python install the necessary depends to
run unittests. I know there are people who would prefer we did that from
OS packages, but that cat is out of the bag and _way_ out of scope for
this particular discussion. Suffice it to say, each project needs a list
of depends that it can hand to pip (or in the case of the client libs,
additionally express in setup.py so that when the client lib is
installed via pip as a dependency of something else, all of its depends
come as well)

So thing number one is that there will be a python requirements.txt
format file in each project. The problem at the point is keeping them in
sync so that we don't have version drift, which is a thing we decided at
ODS that we wanted to do.

When I mentioned devstack in my earlier email, what I was referring to
is the fact that, since devstack installs each of the projects in
sequence, although we do not at the moment have a de jure global
dependency list, in practice we run all of our integration testing on
installs on a single machine - which means there _is_ a single set of
packages and versions that everything is required to run from - we just
don't declare it.

Back to the specific implementation. There are three main ways we've
come up with to solve having this list. The first had to do with git
submodules. We discarded this because most people are still against the
use of submodules in the project, and also because it makes testing
coordinated changes quite difficult here. (if you want to increase the
version of webob that the project uses, do you need to land that change
in the depend list and support it in multiple projects that are
consuming that list) The other problem is that if we just made the tools
dir a submodule (or something similar) there would be no way to exclude
certain depends from certain projects. The total list is quite large,
and for many of our projects (swift and all of the client libs) it would
introduce a massive additional unnecessary installation burden for unit
testing.

The second and third both involve copying, although the mechanism could
be different. In one version, we would do as you suggest above and post
the global depends file up on a website somewhere. Then, at runtime, the
project could do a wget of the file, perform the merge into the local
requirements list, and then go on about its business. It means that the
merge code would run hundreds of time more, the projects would lose the
ability to control when they sync up with the global depends, and it
would make it much more brittle to use any of our projects. So we
thought that since we've already gone down the hell-hole of copying code
from openstack-common, copying requirements file contents at the same
time won't kill us.

Is that more helpful?

Thanks!
Monty

> On 7/2/12 3:40 PM, "Monty Taylor" <mordred@xxxxxxxxxxxx
> <mordred@xxxxxxxxxxxxx>> wrote:
> 
>     Hey all!
> 
>     One of the tasks from the last ODS was to implement a single global
>     dependency list. Turns out the more you think about it, the more
>     important it is... because of the way we use devstack as part of the
>     gate, we actually _currently_ have a de facto global dependency list,
>     it's just not declared anywhere. (oops)
> 
>     Anyway - the original thought was to put the depends in
>     openstack-common. We'd use update.py to copy the depends in to the
>     project, so that projects could align on their own timeframe.
>     Additionally, we'd make the copy only copy in the versions from
>     openstack-common for package that were already listed in the target
>     project, so that we wouldn't add django to python-swiftclient, for
>     instance..
> 
>     The mechanics of that all work and are ready - but then bcwaldon pointed
>     out that it didn't make a ton of sense for them to go in
>     openstack-common, since that has its own lifecycle and is a place for
>     common code to go - not just a catch all place.
> 
>     To that end, I took the code we had written for the update logic and put
>     it, along with the depends lists, into its own repo. I think we're ready
>     to start actually moving forward with it - but we've run up against the
>     hardest problem we every have:
> 
>     naming
> 
>     openstack-depends already got vetoed on IRC because it makes people
>     think of adult diapers. I'm proposing openstack-requires, since the
>     files we're talking about are actually python requirements files.
> 
>     Any objections?
> 
>     We've also been discussing an idea that we could write some gating tests
>     that are only run against milestone-proposed branches that would verify
>     that the requires files in a given project match the versions in
>     openstack-requires - that way there is some lee-way throughout the
>     cycle, but the expectation is that by release time the requires files
>     will be brought in line with the rest of the project. (it's an option if
>     people find that useful - certainly not required)
> 
>     Finally, as an added bonus to this approach, once we have the list and
>     we're even mostly aligned on it, since a library version would need to
>     be in openstack-requires before it hits a project, we can use it as the
>     main basis for our pypi mirror and turn off the upstream pypi source
>     from our jenkins slaves. This would remove the last of the ephemeral
>     build errors that are due to upstream network timeouts. I'm sure
>     everyone will be pleased about that.
> 
>     Anyway - I think general buy in on at _least_ the name
>     openstack-requires will let us move forward with the next step. After
>     that point, it'll all be normal code reviews.
> 
>     Monty
> 
>     _______________________________________________
>     Mailing list: https://launchpad.net/~openstack
>     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>     Unsubscribe : https://launchpad.net/~openstack
>     More help   : https://help.launchpad.net/ListHelp
>