openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Hi Stackers,

tl;dr
-----

There are myriad Chef cookbooks "out there" in the ecosystem and locked 
up behind various company firewalls. It would be awesome if we could 
agree to:

* Align to a single origin repository for OpenStack cookbooks
* Consolidate OpenStack Chef-based deployment experience into a single 
knowledge base
* Have branches on the origin OpenStack cookbooks repository that align 
with core OpenStack projects
* Automate the validation and testing of these cookbooks on multiple 
supported versions of the OpenStack code base

Details
-------

Current State of Forks
======================

Matt Ray and I tried to outline the current state of the various 
OpenStack Chef cookbooks this past Thursday, and we came up with the 
following state of affairs:

** The "official" OpenStack Chef cookbooks **

https://github.com/openstack/openstack-chef

These chef cookbooks are the ones maintained mostly by Dan Prince and 
Brian Lamar and these are the cookbooks used by the SmokeStack project. 
The cookbooks contained in the above repo can install all the core 
OpenStack projects with the exception of Swift and Horizon.

This repo is controlled by the Gerrit instance at review.openstack.org 
just like other core OpenStack projects.

However, these cookbooks DO NOT currently have a stable/diablo branch -- 
they are updated when the development trunks of any OpenStack project 
merges a commit that requires deployment or configuration-related 
changes to their associated cookbook.

Important note: it's easy for Dan and Brian to know when updates to 
these cookbooks are necessary -- SmokeStack will bomb out if a 
deployment-affecting configuration change hits a core project trunk :)

These cookbooks are the ONLY cookbooks that contain stuff for deploying 
with XenServer, AFAICT.

** NTT PF Lab Diablo Chef cookbooks **

https://github.com/ntt-pf-lab/openstack-chef/

So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011, 
because they needed a set of Chef cookbooks for OpenStack that 
functioned for the Diablo code base.

While Nov 11, 2011, is not the *exact* date of the Diablo release, these 
cookbooks do in fact work for a Diablo install -- Nati Ueno is using 
them for the FreeCloud deployment so we know they work...

** OpsCode OpenStack Chef Cookbooks **

Matt Ray from OpsCode created a set of cookbooks for OpenStack for the 
Cactus release of OpenStack:

https://github.com/mattray/openstack-cookbooks
http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef

These cookbooks were forked from the Anso Labs' original OpenStack 
cookbooks from the Bexar release and were the basis for the Chef work 
that Dell did for Crowbar. Crowbar was originally based on Cactus, and 
according to Matt, the repositories of OpenStack cookbooks that OpsCode 
houses internally and uses most often are Cactus-based cookbooks. (Matt, 
please correct me if I am wrong here...)

** Rackspace CloudBuilders OpenStack Chef Cookbooks **

The RCB team also has a repository of OpenStack Chef cookbooks:

https://github.com/cloudbuilders/openstack-cookbooks

Now, GitHub *says* that these cookbooks were forked from the official 
upstream cookbooks, but I do not think that is correct. Looking at this 
repo, I believe that this repo was *actually* forked from the Anso Labs 
OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical.

** Anso Labs OpenStack Chef Cookbooks **

These older cookbooks are in this repo:

https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks

Interestingly, this repo DOES contain a cookbook for Swift.

Current State of Documentation
==============================

Documentation for best practices on using Chef for your OpenStack 
deployments is, well, a bit scattered. Matt Ray has some good 
information on the README on his cookbook repo and the OpsCode wiki:

https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md
http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef

But it is unfortunately not going to help people looking to deploy 
Diablo and later versions of OpenStack.

Most of the other repos contain virtually no documentation on using the 
cookbooks or how they are written.

I have a suspicion that one of the reasons that there has been such a 
proliferation of cookbooks has been the lack of documentation pointing 
people to an appropriate repo, how to use the cookbooks properly, and 
what the best practices for deployment are. That, and the fact that 
folks are just trying to stand up complex clouds and Get Things Done, 
and documentation is annoying to write ;)

Proposal for Alignment
======================

I think the following steps would be good to get done by the time Essex 
rolls out the door in April:

1) Create a stable/diablo branch of the openstack/openstack-chef 
cookbook repo and maintain it in the same way that we maintain stable 
branches for core OpenStack projects. I propose we use the branch point 
that NTT PF Lab used to create their fork of the upstream repo.

2) Work with Matt Ray and other Chef experts to combine any and all best 
practices that may be contained in the non-official cookbook repos into 
the upstream official repository. From a cursory overview, there are 
some differences in how databags are handled, how certs are handled, how 
certain cookbooks are constructed, and of course differences in the 
actual cookbooks in the repos themselves.

3) Consolidate documentation on how to use the cookbooks, the best 
practices used in constructing the cookbooks, and possibly some 
videos/tutorials walking folks through this critical piece of the 
OpenStack puzzle.

4) Create Jenkins builders for stable branch deployment testing. We 
currently test the official development cookbooks by way of SmokeStack 
gates on all core OpenStack projects. Would be great to get the same 
testing automated for non-development branches of the cookbooks.

Thoughts and criticism most welcome, and apologies in advance if I got 
any of the above history wrong. Feel free to correct me!

Best,
-jay