openstack team mailing list archive

[Joshua Harlow <harlowja@xxxxxxxxxxxxx>]

Sure, that would make sense, lets see where the next meeting takes us.

Nothing is ever in stone when its software :-P

On 10/26/12 3:08 AM, "Doug Hellmann" <doug.hellmann@xxxxxxxxxxxxx> wrote:

>
>On Oct 25, 2012, at 9:44 PM, Joshua Harlow <harlowja@xxxxxxxxxxxxx> wrote:
>
>> As for statgen, I think that¹s just a temp repo, it'd be nice to have
>>the
>> end result of this be a library that provides somewhat generic metrics
>>and
>> plugins and such so that stacktech could use the outputs of it,
>>ceilometer
>> could the outputs and other systems could use the outputs (where an
>>output
>> goes would be configurable so that each system can configure its outputs
>> as the operator desires, ie I want my MONITOR metrics to go to MQ in
>> ceilomter and stacktech consumable formats, or to files or to...).
>> 
>> I think when this gets going we should have some repo/project name that
>> goes on stackforge so that even while this is being developed it can go
>> through the normal review process and such (and not in someones special
>> github repo). But we have to start somewhere, something we can discuss
>>as
>> to what a good solution is @ the meeting.
>
>At the summit, as part of the discussions around expanding the scope of
>ceilometer to cover measurement for monitoring, we discussed developing
>the library as part of ceilometer for now and either moving it to Oslo
>for release as a library or just releasing the library as a separate
>package from the ceilometer project directly.
>
>Doug
>
>> 
>> -Josh 
>> 
>> On 10/25/12 5:47 PM, "Jeffrey Budzinski" <jeffreyb@xxxxxxxxxxxxx> wrote:
>> 
>>> 
>>> Yes, I think support for metrics objects that can be leveraged both by
>>> monkey patches and decorators was what we'd been thinking along the
>>>lines
>>> of. The metrics would be controlled via config both in what scopes are
>>> active (e.g. on|off for a package, module, etc.) and also the outlet
>>>for
>>> the metrics (file, datagram, rpc). Support for instrumentation levels
>>> would also be nice so that metric flow could be controlled (i.e.
>>> instrumentation point is annotated as METRIC, MONITOR, PROFILE and then
>>> level to actually emit can be set in conjunction with a metric
>>> outlet/sink). With this approach, folks could control both the volume
>>>of
>>> metrics and also the outlet for the metrics. Ceilometer would also be
>>>an
>>> outlet that could be leveraged via RPC flow for metrics -- though I'd
>>> expect some would want to send via datagram to statsd or file for
>>>offline
>>> log aggregation.
>>> 
>>> I'll post a diagram tomorrow for review and comment.
>>> 
>>> Oh btw, I updated the spec with most of what was in the etherpad. We
>>>can
>>> update the spec to reflect whatever we decide is the preferred
>>>approach.
>>> 
>>> -jeff
>>> 
>>> On Oct 25, 2012, at 5:30 PM, Angus Salkeld wrote:
>>> 
>>>> On 25/10/12 11:13 +0000, Sandy Walsh wrote:
>>>>> grizzly-common-instrumentation seems to be the best choice ...
>>>>> hopefully the other groups will use this etherpad too.
>>>>> 
>>>>> We need a proper blueprint to nail down the approach. IRC is great,
>>>>> but doesn't retain history for other groups. I think we need to get a
>>>>> plan for translating the etherpad into something concise and nailed
>>>>> down.
>>>> 
>>>> Agree.
>>>> 
>>>>> 
>>>>> statgen should really just be a new notifier in Tach (or Scrutinize)
>>>>> ... vs copy-pasting the code into yet-another repo.  Hopefully that's
>>>>> the plan? Tach should remain a generic tool and not pegged to
>>>>>OpenStack.
>>>> 
>>>> Well that was just an "ideas play pen" not serious code.
>>>> 
>>>> I might be coming at this from a slightly different angle...
>>>> I was looking at a library that can be used to generate trace,
>>>> monitoring
>>>> and metering data (kind of like log levels for logging). Currently
>>>>both
>>>> Tach and Scrutinize don't have enough fields (of course that could be
>>>> changed).
>>>> 
>>>> Also I think we should be able to insert instrumentation into the code
>>>> as well
>>>> as have the function level performance metrics monkey patched.
>>>> 
>>>> Then we could have a config that directed metric data to different
>>>> notifiers
>>>> like how you do it in Scrutinize perhaps. Also enforcing data rate
>>>> limits
>>>> and possible data aggregation could be neat configurable features.
>>>> 
>>>> Anyway more at the meeting...
>>>> 
>>>> -Angus
>>>> 
>>>>> 
>>>>> -S
>>>>> ________________________________________
>>>>> From: openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx
>>>>> [openstack-bounces+sandy.walsh=rackspace.com@xxxxxxxxxxxxxxxxxxx] on
>>>>> behalf of Angus Salkeld [asalkeld@xxxxxxxxxx]
>>>>> Sent: Thursday, October 25, 2012 1:00 AM
>>>>> To: openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Subject: Re: [Openstack] Ceilometer, StackTach, Tach / Scrutinize,
>>>>> CloudWatch integration ... Summit followup
>>>>> 
>>>>> On 24/10/12 23:35 +0000, Sandy Walsh wrote:
>>>>>> Hey y'all,
>>>>>> 
>>>>>> Great to chat during the summit last week, but it's been a crazy few
>>>>>> days of catch-up since then.
>>>>>> 
>>>>>> The main takeaway for me was the urgent need to get some common
>>>>>> libraries under these efforts.
>>>>> 
>>>>> Yip.
>>>>> 
>>>>>> 
>>>>>> So, to that end ...
>>>>>> 
>>>>>> 1. To those that asked, I'm going to get my slides / video
>>>>>> presentation made available via the list. Stay tuned.
>>>>>> 
>>>>>> 2. I'm having a hard time following all the links to various efforts
>>>>>> going on (seems every time I turn around there's a new
>>>>>> metric/instrumentation effort, which is good I guess :)
>>>>> 
>>>>> Here is some fun I have been having with a bit of tach+ceilometer
>>>>>code.
>>>>> https://github.com/asalkeld/statgen
>>>>> 
>>>>>> 
>>>>>> Is there a single location I can place my feedback? If not, should
>>>>>>we
>>>>>> create one? I've got lots of suggestions/ideas and would hate to
>>>>>>have
>>>>>> to duplicate the threads or leave other groups out.
>>>>> 
>>>>> I'll add some links here that I am aware of:
>>>>> https://bugs.launchpad.net/ceilometer/+bug/1071061
>>>>> https://etherpad.openstack.org/grizzly-common-instrumentation
>>>>> https://etherpad.openstack.org/grizzly-ceilometer-actions
>>>>> 
>>>>> 
>>>>>https://blueprints.launchpad.net/nova/+spec/nova-instrumentation-metri
>>>>>cs
>>>>> -monitoring
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 3. I'm wrapping up the packaging / cleanup of StackTach v2 with
>>>>>> Stacky and hope to make a more formal announcement on this by the
>>>>>>end
>>>>>> of the week. Lots of great changes to make it easier to use/deploy
>>>>>> based on the Summit feedback!
>>>>>> 
>>>>>> Unifying the stacktach worker (consumer of events) into ceilometer
>>>>>> should be a first step to integration (or agree upon a common
>>>>>> YAGI-based consumer?)
>>>>>> 
>>>>>> 4. If you're looking at Tach, you should also consider looking at
>>>>>> Scrutinize (my replacement effort)
>>>>>> https://github.com/SandyWalsh/scrutinize (needs packaging/docs and
>>>>>> some notifier tweaks on the cprofiler to be called "done for now")
>>>>> 
>>>>> Looks great! I like the monkey patching for performance as you have
>>>>> done here, but we also need a nice clean way of manually inserting
>>>>> instrumentation
>>>>> too (that is what I have been experimenting with in statgen).
>>>>> 
>>>>> Can we chat in #openstack-metering so we are a bit more aware what we
>>>>> are all up to?
>>>>> 
>>>>> 
>>>>> -Angus
>>>>> 
>>>>>> 
>>>>>> Looking forward to moving ahead on this ...
>>>>>> 
>>>>>> Cheers,
>>>>>> -S
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>> 
>>> 
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>> 
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp