openstack team mailing list archive

[Doug Hellmann <doug.hellmann@xxxxxxxxxxxxx>]

On Fri, May 25, 2012 at 4:55 AM, Szymon Grzybowski <semyazz@xxxxxxxxx>wrote:

> Hi,
>
> We had a plan to write some "plugin" which will collect metering data from
> VMs and Hosts (free ram, networking, disk IO, cpu both from VMs and Hosts)
> via libvirt (mayby later for xen etc), but i've found ceilometer project,
> which is doing what I want or will be doing what I want in near future :).
> I was thinking about collectd/ganglia/munin as source of data and I wanted
> to write local agents which will poll those daemons (collectd/ganglia) and
> push data to Rabbit, so it's pretty close to ceilometer's guys.
>

The idea with metering is that by standardizing on a message format you can
tie in at a couple of different points. For publishing data, you could
write a plugin for our agent that does the polling and lets the agent
publish, or you could have your agent publish messages directly on the
metering queue in the right format. However, our focus right now is on
capturing information about utilization so we can document it and charge
the customer. For the sort of features you are talking about, you probably
want more data collected more frequently.


> My main purpose is to consume those messages and write algorithm to manage
> cloud from administrative perspective. I'd like to create something like
> DRM in VmWare - I was on presentation about this topic recently and from
> that brief I learnt that VmWare has something like rules to manage VMs. I
> think, that example (use case) from presentation would be the best way to
> describe what I'm trying to point out:
>
> 1. Administrator defined a rule: { If free ram on host is less than 15%,
> send notification "Alert less than 15% free ram on host"}
> 2. HostA hits less than 15% of free ram and sends notification "Alert less
> than 15% free ram on HostA".
> 3. Central collector or central daemon receives Alert about free ram.
> 4. Central collector pick VM from HostA and migrate it to HostB. - How he
> decides which VM and to which Host migrate is part of algorithm/policy.
>
> This is the basic idea, how does it work - this is what I've found out
> during presentation. And now my whole idea about this mechanism and
> openstack: there could be different policies(algorithms/plugins) how, where
> and when migrate VMs and this could be implemented in external plugins.
>
> Someone can say that there is nova-scheduler. This scenario isn't exactly
> what nova-scheduler is doing, but I see, that in this solution
> nova-scheduler can play his role as service, which picks best host to
> migrate VM.
>

It sounds like an updated scheduler class could use this new data to make
its decision about which host is least busy, at least.


> I'm trying to find out what can I use from ceilometer as metering system.
> I've read few pages on wiki and posts on mailing list about ceilometer's
> architecture, messages etc.
> Right now I know there are Agents and Collectors. Agents get data from
> hypervisor (metering data)  and push them in queue (nova notification
> system). Collectors listens to queue's topic and receives those messages.
> I'd like to:
>

The agent will publish to a new queue using a pattern similar to
notification, but the messages won't be in quite the same format or on the
same exchange.

>
>    - write plugin for agent to send proper alert when metering data hits
>    proper rules (just like in above scenario with 15% free ram)
>
>
>    - write service which will consume this alert (listen proper topic in
>    queue) and fetch data from ceilometer collector?? (or it's backend via API)
>    and find out which VM should be migrated and where (or delegate "Where" to
>    nova-scheduler)
>
>

>
>    -
>
> Another idea. Are there any Host states in Openstack? For example
> "Maintenance state"? Besides scenarios with free ram, cpu and other
> resources there is also typical example of how useful this mechanism could
> be with maintenance state.
> *Scenario:*
> 1. Administrator changes HostA state to "maintenance".
> 2. There goes alert "HostA maintenance".
> 3. Central collector or central daemon receives Alert about maintenance.
> 4. Central collector gets list of VMs on HostA.
> 5. Central collectors start migration all VMs from HostA to other Hosts
> (or invoke nova-scheduler to reassign VM from HostA to other hosts, but in
> this case we need to change or implement new filter in nova-scheduler)
> 6. Administrator changes HostA state to "active".
> 7. Openstack starts new machines on this node (this is now done via
> nova-scheduler) or migrate VMs from other overloaded hosts to HostA.
>
> Everything should be event-driven (alert-notification, host-state-change,
> administrator's action).
>
> I'm also wondering what about this ResourceMonitorAlertsandNotifications<http://wiki.openstack.org/ResourceMonitorAlertsandNotifications>,
> because when I'm looking at schema, it has everything what I need, but
> blueprint link is broken. Is this idea obsolete? Or is it implemented?
> (this was created 2nd April 2012, so I think it's not implemented yet).
>
> First question: Does it make sense? Are such mechanisms implemented in
> Openstack right now? Or is it worth to implement?
>

The feature you are planning sounds interesting, and the implementation for
the data collection will follow a pattern similar to what we are doing for
metering. I don't know if it will make sense to implement the new
collection as part of the metering system for two reasons. First, the
frequency we plan to collect data may be too low for your purposes. That is
adjustable, so maybe it can work. Second, the metering data is meant to be
used for billing, so I'm not sure resource availability information like
RAM on the host fits into the model. Again, that is flexible and open to
re-interpretation.


> Second question (if first's answer isn't negative): How and when can I use
> ceilometer to do what I've described? From meeting's topics<http://wiki.openstack.org/Meetings/MeteringAgenda>I know that yesterday you were deciding what to use as notification bus,
> and in june you will be thinking about storage backend, how is it going on
> right now? Is such scenario possible in ceilometer (I mean plugins in
> agents)? Collecting data not only from guests but also from hosts.
>

The ceilometer agent runs on the host, so it can see the information you
want. However, given the limitations I described above, it may make more
sense to set up a "monitoring" system that performs operations similar to
ceilometer but uses its own messaging exchange so the messages can be
routed to the right listeners. Or perhaps the ceilometer agent should
become a more general-purpose tool that emits information for both types of
consumers.

Doug