openstack team mailing list archive

[Doug Hellmann <doug.hellmann@xxxxxxxxxxxxx>]

[copying the list]

On Tue, May 22, 2012 at 5:02 PM, Matt Joyce <matt.joyce@xxxxxxxxxxxxxxxx>wrote:

>  If the agent is simply passively passing data up stream to the collectors
> I really don't care.  As long as it is never accepting commands remotely.
>
> Once we do that it becomes something else.  Either it's tied into an API
> or it's acting as an independent agent of execution.
>
> Netstack / quantum folks NEED a highly dynamic remote agent they can pass
> execution parameters to.  And they will build one.
>
> But if we keep spinning off these types of directly addressable agents of
> execution we are creating a lot of new things we need to maintain and vet.
>
> If we have a single agent of execution project we can pool development
> efforts and focus on hardening it for all.
>
> That would be my primary concern in the agent.


That makes sense. In our case we have not identified a need for the agent
to receive commands. We are using the nova.service module to run the daemon
right now, so although we have created a new daemon we haven't done a lot
of work to "invent" anything new.

If there is other work going on to create a single daemon to run on the
compute node and if we can tap into that daemon and ensure that periodic
tasks are triggered at regular (and relatively dependable) intervals then
we could move the agent portion of ceilometer into the common agent.

Is there a project for creating that? Or a blueprint I could subscribe to?


>
>
> -Matt
>
>
> On Tue, May 22, 2012 at 11:08 AM, Doug Hellmann <
> doug.hellmann@xxxxxxxxxxxxx> wrote:
>
>>
>>
>> On Tue, May 22, 2012 at 12:53 PM, Matt Joyce <matt.joyce@xxxxxxxxxxxxxxxx
>> > wrote:
>>
>>> My point of concern.
>>> \
>>> If an agent is being built into the compute nodes, that would best be a
>>> split out project.
>>>
>>> Two major reasons.  First and foremost sub projects should not be
>>> spinning up their own agents.  Secondly, there is a use case of agents
>>> outside of metering.
>>>
>>> If an agent is to be built it is a not insignificant change in
>>> architecture for openstack.
>>
>>
>> This agent will run on the compute host, next to nova-compute, not inside
>> the VM. Is that still a concern?
>>
>>
>>>
>>>
>>> -Matt
>>>
>>>
>>> On Tue, May 22, 2012 at 7:30 AM, Doug Hellmann <
>>> doug.hellmann@xxxxxxxxxxxxx> wrote:
>>>
>>>>
>>>>
>>>> On Tue, May 22, 2012 at 4:05 AM, Endre Karlson <endre.karlson@xxxxxxxxx
>>>> > wrote:
>>>>
>>>>> If I'm understanding this correctly, the Collector is kind of like a
>>>>> Agent in Qantum (It sits on a machine doing stuff and passing info
>>>>> upstream).
>>>>>
>>>>> If you look at the approach they have now in Quantum Agent it's
>>>>> writing directly to the DB. But looking at the next version they seem to be
>>>>> moving to having the Agent send data upstream to the Plugin in Quantum. Why
>>>>> not do something similar?
>>>>>
>>>>> I mean if you have a MQ cluster in a deployment I think it makes more
>>>>> sense to have 1 thing that handles the db stuff then having each Collector
>>>>> connect to the db..
>>>>>
>>>>
>>>> That was the goal, but I may have swapped the terminology around. For
>>>> ceilometer, the "agent" runs on the compute node and writes only to the
>>>> message queue. The "collector" runs in a central location and writes to the
>>>> database. The number of collectors you need will depend on the number of
>>>> messages being generated, but the architecture supports running several in
>>>> parallel in a way that each instance does not need to be aware of the
>>>> others.
>>>>
>>>> Doug
>>>>
>>>>
>>>>> Endre.
>>>>> 2012/5/22 Nick Barcet <nick.barcet@xxxxxxxxxxxxx>
>>>>>
>>>>>> On 05/21/2012 10:52 PM, Doug Hellmann wrote:
>>>>>> > I have written up some of my thoughts on a proposed design for
>>>>>> > ceilometer in the wiki [1]. I'm sure there are missing details, but
>>>>>> I
>>>>>> > wanted to start getting ideas into writing so they could be
>>>>>> discussed
>>>>>> > here on the list, since I've talked about different parts with a
>>>>>> couple
>>>>>> > of you separately.
>>>>>> >
>>>>>> > Let me know what you think, and especially if I am not clear or have
>>>>>> > left out any details.
>>>>>> >
>>>>>> > Thanks,
>>>>>> > Doug
>>>>>> >
>>>>>> > [1]
>>>>>> http://wiki.openstack.org/EfficientMetering/ArchitectureProposalV1
>>>>>>
>>>>>> Thanks a lot for putting this together Doug.
>>>>>>
>>>>>> A few questions:
>>>>>>
>>>>>> * "The collector runs on one or more central management servers to
>>>>>> monitor the message queues (for notifications and for metering data
>>>>>> coming from the agent). Notification messages are processed and turned
>>>>>> into metering messages and sent back out onto the message bus using
>>>>>> the
>>>>>> appropriate topic. Metering messages are written to the data store
>>>>>> without modification."
>>>>>> -> Is the reason behind why collectors do not write directly to the
>>>>>> database a way to allow db less implementations as Francis suggested
>>>>>> earlier?  In this case it may be useful to say it explicitly.
>>>>>>
>>>>>> * "Plugins may require configuration options, so when the plugin is
>>>>>> loaded it is asked to add options to the global flags object, and the
>>>>>> results are made available to the plugin before it is asked to do any
>>>>>> work."
>>>>>> -> I am not sure where the "global flags object" resides and how
>>>>>> option
>>>>>> are populated.  I think it would make sense for this to be globally
>>>>>> controlled, and therefore may require for a simple discovery exchange
>>>>>> on
>>>>>> the queue to retrieve values and set defaults if it does not exist
>>>>>> yet.
>>>>>>
>>>>>> * "Metering messages are signed using the hmac module in Python's
>>>>>> standard library. A shared secret value can be provided in the
>>>>>> ceilometer configuration settings. The messages are signed by feeding
>>>>>> the message key names and values into the signature generator in
>>>>>> sorted
>>>>>> order. Non-string values are converted to unicode and then encoded as
>>>>>> UTF-8. The message signature is included in the message for
>>>>>> verification
>>>>>> by the collector."
>>>>>> -> The signature is also kept in the database for future audit
>>>>>> processes, maybe worth mentioning it here.
>>>>>> -> In addition to a signature, I think we would need a sequence number
>>>>>> to be embedded by the agent for each message sent, so that loss of
>>>>>> messages, or forgery of messages, can be detected by the collector and
>>>>>> further audit process.
>>>>>>
>>>>>> Thanks again,
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>