openstack team mailing list archive

[Thierry Carrez <thierry@xxxxxxxxxxxxx>]

Sascha Peilicke wrote:
> Currently, the majority of OpenStack components make use of the
> Python module "setuptools_git" in order to install additional
> configuration files. This is basically the same functionality that
> the MANIFEST.in file (setuptools/distribute) provides, but
> automatic.

Note: This is a development topic, it should (have) be(en) posted to
openstack-dev to reach the appropriate audience. Please follow-up there.

> However, we recently discovered that this approach has issues from
> a packaging perspective. We weren't getting all the data/config
> files that the python package usually gets even though we were
> running the same commands:
> 
> $ python setup.py build
> 
> followed by:
> 
> $ python setup.py install --skip-build
> 
> We are building RPM packages from release tarballs (such as [1]),
> which of course don't include the .git directory. Therefore the
> setuptools_git approach can't do its magic, thus our package builds
> get wrong results. Having OpenStack components rely on
> setuptools_git at build time means we have to distribute the whole
> git repository along with the source code tarball. Of course this
> makes no sense, since it would increase the size of release
> tarballs dramatically and won't get shipped in distributions
> anyway.Therefore, we (and potentially other distribution vendors)
> would have to track these files manually in our RPM spec files. 
> Some reviews have already been opened on the topic (admittedly
> before we discovered the real issue). Given the different outcome
> of each review it seems that not everybody is aware that
> setuptools_git is used or of what it does.
> 
> https://review.openstack.org/#/c/17122/ (ceilometer) - this one
> got accepted before we knew what was going on
> 
> https://review.openstack.org/#/c/17347/ (cinder) - abandoned until
> the situation is clarified
> 
> https://review.openstack.org/#/c/17355/ (nova) - rejected
> 
> So the better solution would be to stop using setuptools_git and
> just include all the data/config files that are meant to be
> distributed in the MANIFEST.in file. This is what every Python
> developer should know about and has the benefit of increased
> transparency about what gets installed and what not. We created a
> blueprint to track this [2].
> 
> Thoughts?

A bit of history here:

We used to rely on MANIFEST.in to list all files, but people routinely
forgot to add new files to it. Apparently "every Python developer"
doesn't know (or care) about this. The end result was that we
discovered very late (sometimes after the release itself) that we
built incomplete tarballs. As a quick search[1] shows, I have
personally filed 27 bugs so far on the subject, so it's not a corner case.

[1] http://bit.ly/TDim7U

Relying on setuptools_git instead allows to avoid that issue
altogether. The projects that adopted it became a non-issue. The
projects that didn't adopt it yet are still a problem. I was about to
push setuptools_git support to projects that don't use it yet.

In summary, I would hate it if we went back to the previous situation.
I'm not personally attached to setuptools_git, but any proposed
replacement solution should keep its simplicity.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack