openstack team mailing list archive

[Mark Collier <mark.collier@xxxxxxxxxxxxx>]

Awesome leadership on this Jay! (And Matt) You should come down to Austin more often :)

Definitely seems like an area with a lot of duplicated effort.


Jay Pipes <jaypipes@xxxxxxxxx> wrote:


Hi Stackers,

tl;dr
-----

There are myriad Chef cookbooks "out there" in the ecosystem and locked
up behind various company firewalls. It would be awesome if we could
agree to:

* Align to a single origin repository for OpenStack cookbooks
* Consolidate OpenStack Chef-based deployment experience into a single
knowledge base
* Have branches on the origin OpenStack cookbooks repository that align
with core OpenStack projects
* Automate the validation and testing of these cookbooks on multiple
supported versions of the OpenStack code base

Details
-------

Current State of Forks
======================

Matt Ray and I tried to outline the current state of the various
OpenStack Chef cookbooks this past Thursday, and we came up with the
following state of affairs:

** The "official" OpenStack Chef cookbooks **

https://github.com/openstack/openstack-chef

These chef cookbooks are the ones maintained mostly by Dan Prince and
Brian Lamar and these are the cookbooks used by the SmokeStack project.
The cookbooks contained in the above repo can install all the core
OpenStack projects with the exception of Swift and Horizon.

This repo is controlled by the Gerrit instance at review.openstack.org
just like other core OpenStack projects.

However, these cookbooks DO NOT currently have a stable/diablo branch --
they are updated when the development trunks of any OpenStack project
merges a commit that requires deployment or configuration-related
changes to their associated cookbook.

Important note: it's easy for Dan and Brian to know when updates to
these cookbooks are necessary -- SmokeStack will bomb out if a
deployment-affecting configuration change hits a core project trunk :)

These cookbooks are the ONLY cookbooks that contain stuff for deploying
with XenServer, AFAICT.

** NTT PF Lab Diablo Chef cookbooks **

https://github.com/ntt-pf-lab/openstack-chef/

So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011,
because they needed a set of Chef cookbooks for OpenStack that
functioned for the Diablo code base.

While Nov 11, 2011, is not the *exact* date of the Diablo release, these
cookbooks do in fact work for a Diablo install -- Nati Ueno is using
them for the FreeCloud deployment so we know they work...

** OpsCode OpenStack Chef Cookbooks **

Matt Ray from OpsCode created a set of cookbooks for OpenStack for the
Cactus release of OpenStack:

https://github.com/mattray/openstack-cookbooks
http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef

These cookbooks were forked from the Anso Labs' original OpenStack
cookbooks from the Bexar release and were the basis for the Chef work
that Dell did for Crowbar. Crowbar was originally based on Cactus, and
according to Matt, the repositories of OpenStack cookbooks that OpsCode
houses internally and uses most often are Cactus-based cookbooks. (Matt,
please correct me if I am wrong here...)

** Rackspace CloudBuilders OpenStack Chef Cookbooks **

The RCB team also has a repository of OpenStack Chef cookbooks:

https://github.com/cloudbuilders/openstack-cookbooks

Now, GitHub *says* that these cookbooks were forked from the official
upstream cookbooks, but I do not think that is correct. Looking at this
repo, I believe that this repo was *actually* forked from the Anso Labs
OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical.

** Anso Labs OpenStack Chef Cookbooks **

These older cookbooks are in this repo:

https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks

Interestingly, this repo DOES contain a cookbook for Swift.

Current State of Documentation
==============================

Documentation for best practices on using Chef for your OpenStack
deployments is, well, a bit scattered. Matt Ray has some good
information on the README on his cookbook repo and the OpsCode wiki:

https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md
http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef

But it is unfortunately not going to help people looking to deploy
Diablo and later versions of OpenStack.

Most of the other repos contain virtually no documentation on using the
cookbooks or how they are written.

I have a suspicion that one of the reasons that there has been such a
proliferation of cookbooks has been the lack of documentation pointing
people to an appropriate repo, how to use the cookbooks properly, and
what the best practices for deployment are. That, and the fact that
folks are just trying to stand up complex clouds and Get Things Done,
and documentation is annoying to write ;)

Proposal for Alignment
======================

I think the following steps would be good to get done by the time Essex
rolls out the door in April:

1) Create a stable/diablo branch of the openstack/openstack-chef
cookbook repo and maintain it in the same way that we maintain stable
branches for core OpenStack projects. I propose we use the branch point
that NTT PF Lab used to create their fork of the upstream repo.

2) Work with Matt Ray and other Chef experts to combine any and all best
practices that may be contained in the non-official cookbook repos into
the upstream official repository. From a cursory overview, there are
some differences in how databags are handled, how certs are handled, how
certain cookbooks are constructed, and of course differences in the
actual cookbooks in the repos themselves.

3) Consolidate documentation on how to use the cookbooks, the best
practices used in constructing the cookbooks, and possibly some
videos/tutorials walking folks through this critical piece of the
OpenStack puzzle.

4) Create Jenkins builders for stable branch deployment testing. We
currently test the official development cookbooks by way of SmokeStack
gates on all core OpenStack projects. Would be great to get the same
testing automated for non-development branches of the cookbooks.

Thoughts and criticism most welcome, and apologies in advance if I got
any of the above history wrong. Feel free to correct me!

Best,
-jay

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp