openstack team mailing list archive

[Monty Taylor <mordred@xxxxxxxxxxxx>]

On 02/06/2012 06:07 PM, Jay Pipes wrote:
> Hi Stackers,
> 
> tl;dr
> -----
> 
> There are myriad Chef cookbooks "out there" in the ecosystem and locked
> up behind various company firewalls. It would be awesome if we could
> agree to:
> 
> * Align to a single origin repository for OpenStack cookbooks
> * Consolidate OpenStack Chef-based deployment experience into a single
> knowledge base
> * Have branches on the origin OpenStack cookbooks repository that align
> with core OpenStack projects
> * Automate the validation and testing of these cookbooks on multiple
> supported versions of the OpenStack code base
> 
> Details
> -------
> 
> Current State of Forks
> ======================
> 
> Matt Ray and I tried to outline the current state of the various
> OpenStack Chef cookbooks this past Thursday, and we came up with the
> following state of affairs:
> 
> ** The "official" OpenStack Chef cookbooks **
> 
> https://github.com/openstack/openstack-chef
> 
> These chef cookbooks are the ones maintained mostly by Dan Prince and
> Brian Lamar and these are the cookbooks used by the SmokeStack project.
> The cookbooks contained in the above repo can install all the core
> OpenStack projects with the exception of Swift and Horizon.
> 
> This repo is controlled by the Gerrit instance at review.openstack.org
> just like other core OpenStack projects.
> 
> However, these cookbooks DO NOT currently have a stable/diablo branch --
> they are updated when the development trunks of any OpenStack project
> merges a commit that requires deployment or configuration-related
> changes to their associated cookbook.
> 
> Important note: it's easy for Dan and Brian to know when updates to
> these cookbooks are necessary -- SmokeStack will bomb out if a
> deployment-affecting configuration change hits a core project trunk :)

I would like to get these to have a stable/diablo branch.

> These cookbooks are the ONLY cookbooks that contain stuff for deploying
> with XenServer, AFAICT.

I think Dan and Brian are also deploying kvm as part of smokestack.

> ** NTT PF Lab Diablo Chef cookbooks **
> 
> https://github.com/ntt-pf-lab/openstack-chef/
> 
> So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011,
> because they needed a set of Chef cookbooks for OpenStack that
> functioned for the Diablo code base.
> 
> While Nov 11, 2011, is not the *exact* date of the Diablo release, these
> cookbooks do in fact work for a Diablo install -- Nati Ueno is using
> them for the FreeCloud deployment so we know they work...

If these are a fork of openstack/openstack-chef, could we perhaps make
these the basis of a stable/diablo branch in openstack-chef?

> ** OpsCode OpenStack Chef Cookbooks **
> 
> Matt Ray from OpsCode created a set of cookbooks for OpenStack for the
> Cactus release of OpenStack:
> 
> https://github.com/mattray/openstack-cookbooks
> http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef
> 
> These cookbooks were forked from the Anso Labs' original OpenStack
> cookbooks from the Bexar release and were the basis for the Chef work
> that Dell did for Crowbar. Crowbar was originally based on Cactus, and
> according to Matt, the repositories of OpenStack cookbooks that OpsCode
> houses internally and uses most often are Cactus-based cookbooks. (Matt,
> please correct me if I am wrong here...)
> 
> ** Rackspace CloudBuilders OpenStack Chef Cookbooks **
> 
> The RCB team also has a repository of OpenStack Chef cookbooks:
> 
> https://github.com/cloudbuilders/openstack-cookbooks
> 
> Now, GitHub *says* that these cookbooks were forked from the official
> upstream cookbooks, but I do not think that is correct. Looking at this
> repo, I believe that this repo was *actually* forked from the Anso Labs
> OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical.
> 
> ** Anso Labs OpenStack Chef Cookbooks **
> 
> These older cookbooks are in this repo:
> 
> https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks
> 
> Interestingly, this repo DOES contain a cookbook for Swift.
> 
> Current State of Documentation
> ==============================
> 
> Documentation for best practices on using Chef for your OpenStack
> deployments is, well, a bit scattered. Matt Ray has some good
> information on the README on his cookbook repo and the OpsCode wiki:
> 
> https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md
> http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef
> 
> But it is unfortunately not going to help people looking to deploy
> Diablo and later versions of OpenStack.
> 
> Most of the other repos contain virtually no documentation on using the
> cookbooks or how they are written.
> 
> I have a suspicion that one of the reasons that there has been such a
> proliferation of cookbooks has been the lack of documentation pointing
> people to an appropriate repo, how to use the cookbooks properly, and
> what the best practices for deployment are. That, and the fact that
> folks are just trying to stand up complex clouds and Get Things Done,
> and documentation is annoying to write ;)
> 
> Proposal for Alignment
> ======================
> 
> I think the following steps would be good to get done by the time Essex
> rolls out the door in April:
> 
> 1) Create a stable/diablo branch of the openstack/openstack-chef
> cookbook repo and maintain it in the same way that we maintain stable
> branches for core OpenStack projects. I propose we use the branch point
> that NTT PF Lab used to create their fork of the upstream repo.

You were reading my mind.

> 2) Work with Matt Ray and other Chef experts to combine any and all best
> practices that may be contained in the non-official cookbook repos into
> the upstream official repository. From a cursory overview, there are
> some differences in how databags are handled, how certs are handled, how
> certain cookbooks are constructed, and of course differences in the
> actual cookbooks in the repos themselves.
> 
> 3) Consolidate documentation on how to use the cookbooks, the best
> practices used in constructing the cookbooks, and possibly some
> videos/tutorials walking folks through this critical piece of the
> OpenStack puzzle.

Yes. PLEASE. Dan has been kind enough to offer to do a skype call with
me to walk me through some of this (and then I keep being too busy to
take him up on it) ... but honestly something that doesn't assume I know
a whole hell of a lot about chef would be great.

> 4) Create Jenkins builders for stable branch deployment testing. We
> currently test the official development cookbooks by way of SmokeStack
> gates on all core OpenStack projects. Would be great to get the same
> testing automated for non-development branches of the cookbooks.

Getting chef-based deploys driven by Jenkins is definitely on my todo
list, and I think that should apply to both stable and development
branch deployment. I also would love to see Smokestack support that as well.

> Thoughts and criticism most welcome, and apologies in advance if I got
> any of the above history wrong. Feel free to correct me!
> 
> Best,
> -jay
>