openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #19149
Re: Blueprint proposal: Drop setuptools_git for including data/config files
Is it possible to generate a MANIFEST.in file using setuptools-git? Would that solve simplicity and efficiency?
Sent from my iPhone
On Dec 4, 2012, at 9:07 PM, "Thierry Carrez" <thierry@xxxxxxxxxxxxx> wrote:
> Sascha Peilicke wrote:
>> Currently, the majority of OpenStack components make use of the
>> Python module "setuptools_git" in order to install additional
>> configuration files. This is basically the same functionality that
>> the MANIFEST.in file (setuptools/distribute) provides, but
>> automatic.
>
> Note: This is a development topic, it should (have) be(en) posted to
> openstack-dev to reach the appropriate audience. Please follow-up there.
>
>> However, we recently discovered that this approach has issues from
>> a packaging perspective. We weren't getting all the data/config
>> files that the python package usually gets even though we were
>> running the same commands:
>>
>> $ python setup.py build
>>
>> followed by:
>>
>> $ python setup.py install --skip-build
>>
>> We are building RPM packages from release tarballs (such as [1]),
>> which of course don't include the .git directory. Therefore the
>> setuptools_git approach can't do its magic, thus our package builds
>> get wrong results. Having OpenStack components rely on
>> setuptools_git at build time means we have to distribute the whole
>> git repository along with the source code tarball. Of course this
>> makes no sense, since it would increase the size of release
>> tarballs dramatically and won't get shipped in distributions
>> anyway.Therefore, we (and potentially other distribution vendors)
>> would have to track these files manually in our RPM spec files.
>> Some reviews have already been opened on the topic (admittedly
>> before we discovered the real issue). Given the different outcome
>> of each review it seems that not everybody is aware that
>> setuptools_git is used or of what it does.
>>
>> https://review.openstack.org/#/c/17122/ (ceilometer) - this one
>> got accepted before we knew what was going on
>>
>> https://review.openstack.org/#/c/17347/ (cinder) - abandoned until
>> the situation is clarified
>>
>> https://review.openstack.org/#/c/17355/ (nova) - rejected
>>
>> So the better solution would be to stop using setuptools_git and
>> just include all the data/config files that are meant to be
>> distributed in the MANIFEST.in file. This is what every Python
>> developer should know about and has the benefit of increased
>> transparency about what gets installed and what not. We created a
>> blueprint to track this [2].
>>
>> Thoughts?
>
> A bit of history here:
>
> We used to rely on MANIFEST.in to list all files, but people routinely
> forgot to add new files to it. Apparently "every Python developer"
> doesn't know (or care) about this. The end result was that we
> discovered very late (sometimes after the release itself) that we
> built incomplete tarballs. As a quick search[1] shows, I have
> personally filed 27 bugs so far on the subject, so it's not a corner case.
>
> [1] http://bit.ly/TDim7U
>
> Relying on setuptools_git instead allows to avoid that issue
> altogether. The projects that adopted it became a non-issue. The
> projects that didn't adopt it yet are still a problem. I was about to
> push setuptools_git support to projects that don't use it yet.
>
> In summary, I would hate it if we went back to the previous situation.
> I'm not personally attached to setuptools_git, but any proposed
> replacement solution should keep its simplicity.
>
> --
> Thierry Carrez (ttx)
> Release Manager, OpenStack
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Follow ups
References