openstack team mailing list archive
Mailing list archive
Re: [CHEF] Aligning Cookbook Efforts
On 02/06/2012 06:07 PM, Jay Pipes wrote:
> Hi Stackers,
> There are myriad Chef cookbooks "out there" in the ecosystem and locked
> up behind various company firewalls. It would be awesome if we could
> agree to:
> * Align to a single origin repository for OpenStack cookbooks
> * Consolidate OpenStack Chef-based deployment experience into a single
> knowledge base
> * Have branches on the origin OpenStack cookbooks repository that align
> with core OpenStack projects
> * Automate the validation and testing of these cookbooks on multiple
> supported versions of the OpenStack code base
> Current State of Forks
> Matt Ray and I tried to outline the current state of the various
> OpenStack Chef cookbooks this past Thursday, and we came up with the
> following state of affairs:
> ** The "official" OpenStack Chef cookbooks **
> These chef cookbooks are the ones maintained mostly by Dan Prince and
> Brian Lamar and these are the cookbooks used by the SmokeStack project.
> The cookbooks contained in the above repo can install all the core
> OpenStack projects with the exception of Swift and Horizon.
> This repo is controlled by the Gerrit instance at review.openstack.org
> just like other core OpenStack projects.
> However, these cookbooks DO NOT currently have a stable/diablo branch --
> they are updated when the development trunks of any OpenStack project
> merges a commit that requires deployment or configuration-related
> changes to their associated cookbook.
> Important note: it's easy for Dan and Brian to know when updates to
> these cookbooks are necessary -- SmokeStack will bomb out if a
> deployment-affecting configuration change hits a core project trunk :)
I would like to get these to have a stable/diablo branch.
> These cookbooks are the ONLY cookbooks that contain stuff for deploying
> with XenServer, AFAICT.
I think Dan and Brian are also deploying kvm as part of smokestack.
> ** NTT PF Lab Diablo Chef cookbooks **
> So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011,
> because they needed a set of Chef cookbooks for OpenStack that
> functioned for the Diablo code base.
> While Nov 11, 2011, is not the *exact* date of the Diablo release, these
> cookbooks do in fact work for a Diablo install -- Nati Ueno is using
> them for the FreeCloud deployment so we know they work...
If these are a fork of openstack/openstack-chef, could we perhaps make
these the basis of a stable/diablo branch in openstack-chef?
> ** OpsCode OpenStack Chef Cookbooks **
> Matt Ray from OpsCode created a set of cookbooks for OpenStack for the
> Cactus release of OpenStack:
> These cookbooks were forked from the Anso Labs' original OpenStack
> cookbooks from the Bexar release and were the basis for the Chef work
> that Dell did for Crowbar. Crowbar was originally based on Cactus, and
> according to Matt, the repositories of OpenStack cookbooks that OpsCode
> houses internally and uses most often are Cactus-based cookbooks. (Matt,
> please correct me if I am wrong here...)
> ** Rackspace CloudBuilders OpenStack Chef Cookbooks **
> The RCB team also has a repository of OpenStack Chef cookbooks:
> Now, GitHub *says* that these cookbooks were forked from the official
> upstream cookbooks, but I do not think that is correct. Looking at this
> repo, I believe that this repo was *actually* forked from the Anso Labs
> OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical.
> ** Anso Labs OpenStack Chef Cookbooks **
> These older cookbooks are in this repo:
> Interestingly, this repo DOES contain a cookbook for Swift.
> Current State of Documentation
> Documentation for best practices on using Chef for your OpenStack
> deployments is, well, a bit scattered. Matt Ray has some good
> information on the README on his cookbook repo and the OpsCode wiki:
> But it is unfortunately not going to help people looking to deploy
> Diablo and later versions of OpenStack.
> Most of the other repos contain virtually no documentation on using the
> cookbooks or how they are written.
> I have a suspicion that one of the reasons that there has been such a
> proliferation of cookbooks has been the lack of documentation pointing
> people to an appropriate repo, how to use the cookbooks properly, and
> what the best practices for deployment are. That, and the fact that
> folks are just trying to stand up complex clouds and Get Things Done,
> and documentation is annoying to write ;)
> Proposal for Alignment
> I think the following steps would be good to get done by the time Essex
> rolls out the door in April:
> 1) Create a stable/diablo branch of the openstack/openstack-chef
> cookbook repo and maintain it in the same way that we maintain stable
> branches for core OpenStack projects. I propose we use the branch point
> that NTT PF Lab used to create their fork of the upstream repo.
You were reading my mind.
> 2) Work with Matt Ray and other Chef experts to combine any and all best
> practices that may be contained in the non-official cookbook repos into
> the upstream official repository. From a cursory overview, there are
> some differences in how databags are handled, how certs are handled, how
> certain cookbooks are constructed, and of course differences in the
> actual cookbooks in the repos themselves.
> 3) Consolidate documentation on how to use the cookbooks, the best
> practices used in constructing the cookbooks, and possibly some
> videos/tutorials walking folks through this critical piece of the
> OpenStack puzzle.
Yes. PLEASE. Dan has been kind enough to offer to do a skype call with
me to walk me through some of this (and then I keep being too busy to
take him up on it) ... but honestly something that doesn't assume I know
a whole hell of a lot about chef would be great.
> 4) Create Jenkins builders for stable branch deployment testing. We
> currently test the official development cookbooks by way of SmokeStack
> gates on all core OpenStack projects. Would be great to get the same
> testing automated for non-development branches of the cookbooks.
Getting chef-based deploys driven by Jenkins is definitely on my todo
list, and I think that should apply to both stable and development
branch deployment. I also would love to see Smokestack support that as well.
> Thoughts and criticism most welcome, and apologies in advance if I got
> any of the above history wrong. Feel free to correct me!