openstack team mailing list archive

[Matt Ray <matt@xxxxxxxxxxx>]

I think Jay did a good job outlining the lineages and scope of the
assorted cookbook efforts so far. My Anso-based fork at
github.com/mattray/openstack-cookbooks was the basis for a few public
and private forks and strictly focused on multi-node deployments of
stable releases. A lot of this went into Dell's efforts with Crowbar
and they've continued to develop for Diablo and future releases while
working with Rackspace Cloud Builders. While I've been working with
them primarily since the Diablo release, I've also done work with
folks who do not want to use Crowbar, and pulling those cookbooks out
of the barclamps hasn't always been straightforward.

After discussing the state of Chef cookbooks out there with Jay and
folks at Dell, I plan on forking off of
github.com/openstack/openstack-chef/tree/stable/diablo once it's there
and getting it synced up with the efforts in the Crowbar cookbooks.
I'll be pushing into Opscode's repo and hopefully sending patches
upstream to both efforts.

My repo will probably not be a good candidate for an upstream provider
since updates may be sporadic since I don't only work on OpenStack. I
won't follow trunk and I'll continue to strictly focus on deploying
the current stable release. It's likely at some point my repo will
pull in support for multiple implementations of the Swift API (Ceph,
Gluster, etc.) and hopefully additional hypervisors and databases and
RHEL support; it's driven by whoever I'm currently working with. I
also plan on unforking the many non-OpenStack cookbooks from the repo
and pushing any changes upstream to help keep things simple.

While the Cactus documentation I wrote is stale
(http://bit.ly/OSChef), I'll update it going forward and try to keep
my repo well-documented as far as what all the pieces are and usage.
On Jay's suggestion I'll keep the content synced between the Opscode
wiki and the README.md. My current #1 priority is getting
knife-openstack fixed, then I'll get working on this again.

Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@xxxxxxxxxxx | (512) 731-2218
Twitter, IRC, GitHub: mattray



On Mon, Feb 6, 2012 at 8:07 PM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
> Hi Stackers,
>
> tl;dr
> -----
>
> There are myriad Chef cookbooks "out there" in the ecosystem and locked up
> behind various company firewalls. It would be awesome if we could agree to:
>
> * Align to a single origin repository for OpenStack cookbooks
> * Consolidate OpenStack Chef-based deployment experience into a single
> knowledge base
> * Have branches on the origin OpenStack cookbooks repository that align with
> core OpenStack projects
> * Automate the validation and testing of these cookbooks on multiple
> supported versions of the OpenStack code base
>
> Details
> -------
>
> Current State of Forks
> ======================
>
> Matt Ray and I tried to outline the current state of the various OpenStack
> Chef cookbooks this past Thursday, and we came up with the following state
> of affairs:
>
> ** The "official" OpenStack Chef cookbooks **
>
> https://github.com/openstack/openstack-chef
>
> These chef cookbooks are the ones maintained mostly by Dan Prince and Brian
> Lamar and these are the cookbooks used by the SmokeStack project. The
> cookbooks contained in the above repo can install all the core OpenStack
> projects with the exception of Swift and Horizon.
>
> This repo is controlled by the Gerrit instance at review.openstack.org just
> like other core OpenStack projects.
>
> However, these cookbooks DO NOT currently have a stable/diablo branch --
> they are updated when the development trunks of any OpenStack project merges
> a commit that requires deployment or configuration-related changes to their
> associated cookbook.
>
> Important note: it's easy for Dan and Brian to know when updates to these
> cookbooks are necessary -- SmokeStack will bomb out if a
> deployment-affecting configuration change hits a core project trunk :)
>
> These cookbooks are the ONLY cookbooks that contain stuff for deploying with
> XenServer, AFAICT.
>
> ** NTT PF Lab Diablo Chef cookbooks **
>
> https://github.com/ntt-pf-lab/openstack-chef/
>
> So, NTT PF Lab forked the upstream Chef cookbooks back in Nov 11, 2011,
> because they needed a set of Chef cookbooks for OpenStack that functioned
> for the Diablo code base.
>
> While Nov 11, 2011, is not the *exact* date of the Diablo release, these
> cookbooks do in fact work for a Diablo install -- Nati Ueno is using them
> for the FreeCloud deployment so we know they work...
>
> ** OpsCode OpenStack Chef Cookbooks **
>
> Matt Ray from OpsCode created a set of cookbooks for OpenStack for the
> Cactus release of OpenStack:
>
> https://github.com/mattray/openstack-cookbooks
> http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef
>
> These cookbooks were forked from the Anso Labs' original OpenStack cookbooks
> from the Bexar release and were the basis for the Chef work that Dell did
> for Crowbar. Crowbar was originally based on Cactus, and according to Matt,
> the repositories of OpenStack cookbooks that OpsCode houses internally and
> uses most often are Cactus-based cookbooks. (Matt, please correct me if I am
> wrong here...)
>
> ** Rackspace CloudBuilders OpenStack Chef Cookbooks **
>
> The RCB team also has a repository of OpenStack Chef cookbooks:
>
> https://github.com/cloudbuilders/openstack-cookbooks
>
> Now, GitHub *says* that these cookbooks were forked from the official
> upstream cookbooks, but I do not think that is correct. Looking at this
> repo, I believe that this repo was *actually* forked from the Anso Labs
> OpenStack Chef Cookbooks, as the list of cookbooks is virtually identical.
>
> ** Anso Labs OpenStack Chef Cookbooks **
>
> These older cookbooks are in this repo:
>
> https://github.com/ansolabs/openstack-cookbooks/tree/master/cookbooks
>
> Interestingly, this repo DOES contain a cookbook for Swift.
>
> Current State of Documentation
> ==============================
>
> Documentation for best practices on using Chef for your OpenStack
> deployments is, well, a bit scattered. Matt Ray has some good information on
> the README on his cookbook repo and the OpsCode wiki:
>
> https://github.com/mattray/openstack-cookbooks/blob/cactus/README.md
> http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef
>
> But it is unfortunately not going to help people looking to deploy Diablo
> and later versions of OpenStack.
>
> Most of the other repos contain virtually no documentation on using the
> cookbooks or how they are written.
>
> I have a suspicion that one of the reasons that there has been such a
> proliferation of cookbooks has been the lack of documentation pointing
> people to an appropriate repo, how to use the cookbooks properly, and what
> the best practices for deployment are. That, and the fact that folks are
> just trying to stand up complex clouds and Get Things Done, and
> documentation is annoying to write ;)
>
> Proposal for Alignment
> ======================
>
> I think the following steps would be good to get done by the time Essex
> rolls out the door in April:
>
> 1) Create a stable/diablo branch of the openstack/openstack-chef cookbook
> repo and maintain it in the same way that we maintain stable branches for
> core OpenStack projects. I propose we use the branch point that NTT PF Lab
> used to create their fork of the upstream repo.
>
> 2) Work with Matt Ray and other Chef experts to combine any and all best
> practices that may be contained in the non-official cookbook repos into the
> upstream official repository. From a cursory overview, there are some
> differences in how databags are handled, how certs are handled, how certain
> cookbooks are constructed, and of course differences in the actual cookbooks
> in the repos themselves.
>
> 3) Consolidate documentation on how to use the cookbooks, the best practices
> used in constructing the cookbooks, and possibly some videos/tutorials
> walking folks through this critical piece of the OpenStack puzzle.
>
> 4) Create Jenkins builders for stable branch deployment testing. We
> currently test the official development cookbooks by way of SmokeStack gates
> on all core OpenStack projects. Would be great to get the same testing
> automated for non-development branches of the cookbooks.
>
> Thoughts and criticism most welcome, and apologies in advance if I got any
> of the above history wrong. Feel free to correct me!
>
> Best,
> -jay