openstack team mailing list archive

[Mark McLoughlin <markmc@xxxxxxxxxx>]

On Tue, 2012-06-19 at 11:25 +0200, Thierry Carrez wrote:
> Mark McLoughlin wrote:
> > One question I had on that is re:
> > 
> >   the ability to release a client library outside of the core project 
> >   release cycle (requests have been coming in to our release manager for
> >   this)
> > 
> > Who were these request from and why? That would help understand what
> > we're trying to do here.
> 
> A bit of history would help.
> 
> Originally the client library projects were independent and were
> released by their author to PyPI. People built their tooling and
> automation

Interested in specific examples ...

> around the capability for PyPI to fetch the version they
> needed (I don't really support that, but apparently that's what most
> devs do). So any time someone needed to access a new feature, they would
> ask the author for a new PyPI release. It resulted in a lot of
> confusion, as the projects on PyPI were maintained by a number of folks
> and some of them were a bit unmaintained.

Yep, and since the client libs are important to the project, it makes
sense for them to be maintained by the project and released by the
project's release team.

> Then we realized that the client libraries were needed by core projects
> (Nova -> Glance, Nova -> Nova zones, Horizon, Glance -> Swift, etc.)
> while they were not "OpenStack". Rather than creating a bunch of new
> core projects, the PPB decided to consider them as "separate release
> deliverables" of the corresponding server project, inheriting its PTL
> and Launchpad project.

Yep, reasonable.

> However they also inherited the release scheme of the server project
> (new version every 6 months), which was (or was not) synced to PyPI
> depending of who was owning the PyPI project. More confusion as PyPI
> rarely contained the just-released version, since releasing to PyPI was
> not handled by OpenStack release management.

I would have thought that only OpenStack release management should be
publishing those libs to pypi now.

> People kept on asking for new PyPI releases to access recent features.
> Sometimes it's just that they need an early version of the client
> somewhere to start integrating it before the milestone hits
> (cinderclient, swiftclient). A release every 6 months is not enough.

Well, we do releases more often that that:

 - milestone releases provide early access to features currently under 
   development

 - stable releases provide a safe source of bugfixes to previous 
   releases

If the issue is integration *within* OpenStack - e.g. Nova needing
cinderclient - then I don't think that should dictate the release
schedule for these libs. We can surely figure out how to do that without
doing more regular official releases of the libs? But first, is that
what we're talking about?

If it's not just about integration within the project, what are the
examples of external projects needing OpenStack to release non-milestone
releases of the client libs more often? What kind of features are they
looking for?

> Should PyPI only get releases ? or milestones ? or dailies ? And PyPI
> does not support pre-versioning (2012.2~DATE) in the same way the server
> does. For example there is no way to version our milestones in PyPI.

I think we'd like to be able to publish milestone releases of the client
libs to pypi, but not have "pip install" install milestones unless they
are explicitly requested? Right?

pypi probably doesn't support that, but do we want to allow pypi's
limitations dictate our schedule either?

> For all those reasons, it appears to be simpler to revisit the decision
> of sharing versioning and release scheme with the server project. The
> client library is a separate project anyway, with its own rewrites, APIs
> etc. Conflating the two (like sharing projects in Launchpad) led to
> confusion.

I have to admit that I'm still a bit confused exactly what problem we're
fixing here - e.g. who exactly is demanding OpenStack do more regular
releases of its client libs and why?

> > In any case, my tl;dr suggestion is:
> > 
> >   1) The client libs get released along with OpenStack releases and 
> >      versioned the same.
> > [...]
> >   2) We also include client libs in milestone releases, for people 
> >      wanting to use the libs against development releases of OpenStack. 
> 
> That's the current situation, and as explained above it puts us in a
> very difficult spot with releasing to PyPI. There is no elegant way for
> client libraries to follow OpenStack Core versioning and be often
> released with a version number that makes sense on PyPI. On the other
> hand, Monty's solution solves that.

Ok, so this *is* all about pypi? I'm starting to understand now :)

> So I'd reverse the question: what would be the killer benefit of
> keeping on doing the same thing ?
> 
> I see only one: clear match between client and supported server version.
> I think that can be documented, or we can use a versioning logic that
> makes that very apparent, while still using a clearly separate release
> scheme for client libraries.

No, that's not what I really care about.

First is that when we do our release every 6 months we include a set of
client libraries in that coordinated release. If someone deploys or
distributes a release of OpenStack, that includes the client libs and
CLIs. Making them part of the coordinated release is the easiest way to
give people confidence that the combination of versions works well.

(How those client lib releases in the coordinated release should be
versioned isn't a huge deal, but having the same version number would be
nice and easy to understand)

Second is that we shouldn't be doing official releases of client libs
that use not-yet-frozen REST API features. Milestone releases,
development snapshots, etc. are fine. But we need to give ourselves
*some* flexibility to fix new APIs during the development cycle.

The way I'm seeing it now is there are three development streams:

  1) bugfixes (i.e. suitable for a stable branch)

  2) new client lib features built on top of existing REST API features 
     (e.g. feature additions to the client libs which were released as 
     part of Essex which do not require REST API features from Folsom)

  3) new client lib features built on top of new REST API features 
     (e.g. the Folsom development stream)

IMHO, if you're J. Random Developer doing 'pip install glanceclient',
you probably want (2).

Particularly risk-averse distros may want (1), since (2) is more risky
to pull updates from.

And for integration within the project, or folks testing the latest
development stream, you want (3).

The project probably can't hope to support all 3 development streams,
but can we do (1) and (2) in a "stable" branch and (3) using our usual
development release cycle?

Would that satisfy everyone but the "particularly risk-averse distros"?

Cheers,
Mark.