← Back to team overview

openstack team mailing list archive

Re: nova/puppet blueprint, and some questions


Overall, I really like the idea of providing support for configuration
management through nova.

A couple of issues strike me on this, though.

Having worked quite a lot with Puppet, my experiences of running it in a
client-server configuration at scale have demonstrated that it is hard.
Generally, I have found that running puppet in a serverless mode is
preferable, distributing the modules and site.pp files through some other

Running Nova in a multi-tenant environment is going to introduce a whole
lot more issues, as this is something which Puppet doesn't support very
well. The issues I'm thinking of are things like supporting custom modules
for different tenants where the module may have information which that
tenant would not want shared with others, or supporting different modules
with overlapping namespaces - something that will happen with multiple

Also, how would one handle different operating systems, distros and/or
images? A well coded Puppet module can usually find its way around those
issues, but so many Puppet modules are not sufficiently well coded (or
widely tested) that there will be either so few modules as to make the
service fairly useless or it would only work with specific

What I would consider may work better would be passing a meta-parameter
with a uri. This uri would be a location (maybe a git repo) where all the
relevant puppet modules (or chef recipes for that matter) could be
downloaded by the instance. Alternatively, this could be passed in using
the configuration drive as outlined earlier. A second meta-parameter could
be included pointing to a location for a site.pp file. It would then be
incumbent upon the image to include an appropriate tool to make use of this
configuration, running puppet in a distributed way. If the tenant needed
specific modules, they could be obtained from the tenant's 'own' repository
without effecting other tenants.

If support for Puppet stored configurations was required, that too would
have to be restricted to a per-tenant (possibly even per-project). This
would make it difficult to support in a secure way using just the standard
Nova database. Possibly a better option would be to provide a custom Puppet
stored config driver (maybe an ENC - I've not used them so not sure what
they would add) that could pull down details from a separate database which
would break the data up by tenant, only allowing a tenant to access their
own data.


On Sat, Jan 28, 2012 at 12:56 AM, Ryan Lane <rlane@xxxxxxxxxxxxx> wrote:

> >> http://bogott.net/misc/osmpuppet.png
> >>
> >> That's what it looks like when you configure an instance using Open
> Stack
> >> Manager, which is WikiMedia's VM management interface.  My main
> priority for
> >> adding puppet support to Nova is to facilitate the creation and control
> of a
> >> GUI much like that one.
> >
> >
> >
> > Can you explain how your solution works now? You want to inject data into
> > the VMs in the proposal, but outside of designating the puppet master,
> all
> > the data for variables and classes should be changes to the puppet
> master,
> > not the instances. That's kind of the whole point of the puppet master.
> >
> > One thing you really seem to want is RBAC for the nova users.
> >
> > How are you getting the names for the recipes into your system? Is that
> sync
> > with what is on the puppet master somehow or you are going to do data
> entry
> > and it's all string matching?
> >
> Right now we are adding puppetvar and puppetclass attributes into LDAP
> nodes. An instance uses cloud-init, installs puppet, points puppet at
> the right puppet master, and does a puppet run. A puppet master waits
> for certificate requests, verifies the instance is in LDAP, then signs
> the certificate. The instance's catalogue is generated by the puppet
> master, based on the LDAP entry.
> >> On 1/26/12 5:03 PM, Andrew Clay Shafer wrote:
> >>
> >>
> >> I'd also like to see more of a service oriented approach and avoid
> adding
> >> tables to nova if possible.
> >>
> >> I'm not sure the best solution is to come up with a generic service for
> >> $configuration_manager for nova core. I'd rather see these implemented
> as
> >> optional first class extensions.
> >>
> >> This sounds intriguing, but I'll plead ignorance here; can you tell me
> >> more about what this would look like, or direct me to an existing
> analogous
> >> service?
> >
> >
> > I don't think there is a good existing example, but I know if the defacto
> > way to add functionality in nova is to add tables to the db, that's the
> path
> > to operational and maintenance hell.
> >
> > That's not just for this integration, but in general.
> >
> > For openstack to become what it should be, Nova shouldn't be a monolithic
> > app on a database.
> >
> > Even if you wanted to run this on the same node, it probably shouldn't be
> > tables in the same database. It should be a separate services with it's
> own
> > db user and scheme then be integrated by apis or maybe adding to wsgi.
> >
> Well, I think part of the problem is there doesn't seem to be any way
> to extend nova core right now. The API is extendable, but actions
> generated by existing API can't be modified without hacking at the
> core code. This includes database code/schema as well.
> Having the tables in the same database isn't problematic as long as we
> ensure there aren't naming conflicts. Often the way to handle this is
> to have a standardized naming prefix.
> > I'm confused how you want to run puppet exactly. The site.pp would
> typically
> > live on the puppet master.
> >
> > Can you explain more about what you are thinking or how your current
> > solution works?
> >
> We are currently using the LDAP approach with a puppet master, but
> it's problematic with what we're actually doing.
> In our implementation everything is controlled via puppet, and our
> puppet repo is kept in a gerrit repo. We have a branch for production
> and one for our pre-production cloned environment. One project
> contains our pre-production clone (testlabs). Every other project is
> meant as a space to build things, and then puppetize them. The
> services tested in other projects are moved into testlabs via a
> cherry-pick, then tested there, then cherry-picked across to
> production.
> Currently all projects use the testlabs puppet branch, because the
> centralized puppet master must run on a single branch. What we want is
> for each project to have their own branch of the repo, so that they
> can test all of their changes in their own branch without needing
> code-review and without needing to possibly clobber other projects.
> The problem we are running into is that each project needs its own
> puppet master (and no, puppet environments don't work), as puppet
> isn't multi-tenant. The good thing is, we don't really *need* a puppet
> master. Since we have all of the manifests in a single repo, we can
> just checkout the entire repo, and have the instance run the puppet
> configuration directly. We can create a gerrit branch on project
> creation, and have all instances configured to pull/push using that
> branch.
> So, what we really want is a way to stick the repo and the node
> configuration into something that can be accessed on instance
> creation.
> > I think it would be sweet if nova and the dashboard (and probably
> keystone
> > too) had a standardized way to add integrated functionality. I don't
> believe
> > nova core should be reimplementing/duplicating functionality and logic in
> > other systems.
> >
> > The goal of interacting with the instances through a shared interface is
> a
> > good one, I'm not against that, I just want to see less deep coupling to
> > accomplish it.
> >
> Yes, a shared interface is what we'd like as well. My main motivation
> for getting this moved into nova is that I want people to be able to
> use the cli tools in addition to web tools. This was also my
> motivation for wanting DNS support added ;).
> >> I may be misunderstanding you, or my blueprint may be unclear.
> Available,
> >> Unavailable, and Default don't refer to the availability of classes on
> the
> >> puppet master; rather, they refer to whether or not a class is made
> >> available to a nova user for a given instance.  An 'available' class
> would
> >> appear in the checklist in my screenshot.  An Unavailable class would
> not.
> >> A 'default' class would appear, and be pre-checked.  In all three cases
> the
> >> class is presumed to be present on the puppet master.
> >
> >
> > I already asked this, but what keeps that in sync with the puppet master?
> >
> Nothing, this is user-defined and global defined based on the
> end-user's knowledge of the puppet repo. The puppet master has no way
> of saying which variables are available, at minimum, since it's
> possible for templates to require variables that aren't defined in
> manifests.
> > Personally, I'd rather see an integration that has a per user
> configuration
> > to a puppet master that stays in sync than the RBAC per module.
> Well, we aren't aiming at RBAC, but rather aiming at giving a user a
> list of classes and variables available in a specific project. Think
> of one project mostly wanting to use the CLI, another using horizon,
> and another using OpenStackManager (my MediaWiki extension). Without
> nova knowing what's available in a project there's no way for each
> interface to display this information.
> >>> I also think managing a site.pp is going to be inferior to providing an
> >>> endpoint that can act as an eternal node tool for the puppet master.
> >>> http://docs.puppetlabs.com/guides/external_nodes.html
> >>
> >> In which case nova would interact directly with the puppet master for
> >> configuration purposes?  (I don't hate that idea, just asking for
> >> clarification.)
> >
> >
> > That's puppet's model. Whether you use a site.pp, or external nodes. I'm
> > unclear how you want to do it. Can you explain how your system works now?
> >>
> As described above, we feel that using a non-centralized puppet system
> is preferable (for us). Of course, we'd like to have broad support.
> >>> One other point, that you might have thought of, but I don't see
> anywhere
> >>> on the wiki is how to handle the ca/certs for the instances.
> >>
> >> I believe this (and your subsequent question) falls under the heading
> of "
> >> Instances are presumed to know any puppet config info they need at
> creation
> >> time (e.g. how to contact the puppet master). "  Important, but outside
> the
> >> scope of this design :)
> >
> >
> > Thinking through this is actually critical for any standardized puppet
> > integration in my opinion. The solution is a prerequisite before
> considering
> > anything else.
> Yes, this is an issue. We'd need to figure this out. In the case where
> you don't have a puppet master, this isn't necessary, but for it to be
> generic, it would need to work. It's possible to sign through puppet's
> API, though, so it should be easy enough to have the puppet driver
> handle that, if needed.
> >> So that I understand your terminology... are extensions like the quotas
> or
> >> floating ips considered 'core nova'?
> >
> >
> > There is not a bright line especially with the way things have evolved to
> > now, but I would say floating IPs should definitely be core
> functionality.
> > Quotas may be debatable, but I think it is defensible, though part of me
> > feels like some of that kind of permission functionality might be better
> > decoupled.
> >
> >> Thanks again for your input!  Clearly it would be best to hash this out
> at
> >> the design summit, but I'm hoping to get at least a bit of coding done
> >> before April :)
> >
> >
> > I hope to be there. I do like the idea, I just want to do what's best for
> > OpenStack.
> >
> Agreed. It may be that some form of hook system, or some other way of
> extending nova without hacking core would be a more appropriate way of
> handling things like this. Of course, an extensions system also makes
> compatibility between vendors much more difficult.
> - Ryan Lane
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp