yahoo-eng-team team mailing list archive

[Armando Migliaccio <1438159@xxxxxxxxxxxxxxxxxx>]

** Tags added: loadimpact

** Changed in: neutron
       Status: Opinion => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1438159

Title:
  Made neutron agents silent by using AMQP

Status in neutron:
  Confirmed

Bug description:
  Problem.: Neutron agents does a lot of periodic task which leads  an rpc call + database transaction, which does not even provide a new information, because nothing changed.
  This behaviour in scale can be called as `DDOS attack`, generally this kind of architecture is bad at scaling and can be considered as an any-pattern.

  Instead of periodic poll, we can leverage the AMQP brokers bind capabilities.
  Neutron has many situation like security group rule change or dvr related changes which needs to be communicated with multiple agents, but usually not with all agent.

  The agent at startup needs to synchronise the as usual, but during the
  sync the agent can subscribe to the interesting events to avoid the
  periodic tasks. (Note.: After the first subscribe loop a second one is
  needed to do not miss changes during the subscribe process ).

  The AMQP queues with 'auto-delete' can be considered as a reliable source of information which does not miss any event notification.
  On connection loss or full broker cluster die the agent needs to re sync everything guarded in this way,
  in these cases, the queue will disappear so the situation easily detectable.

  1. Create a Direct exchange for all kind of resources<type> what needs
  to be synchronised in this way, for example.: 'neutron.securitygroups'
  . The exchange declaration needs to happen at q-svc start-up time or
  at full broker cluster die (not-found exception will tell it). The
  exchange SHOULD NOT be redeclared or verified at every message
  publish.

  2. Every agent creates a dedicated per agent queue with auto-delete flag, if the agent already maintains a queue with this property he MAY reuse that one. The agents SHOULD avoid to creating multiple queues per resource type. The messages MUST contain a type information.
  3. All agent creates a binding between his queue and the resource type queue with he realise he depends on the resource, for example it maintains at least one port with the given security-group. (The agents needs to remove the binding. when they stop using it)
  4. The q-svc publishes just a single message  when the resource related change happened. The routing key is the uuid.

  Alternatively a topic exchange can be used, with a single  exchange.
  In this case the routing keys MUST contains the resource type like: neutron.<resource_type>.<uuid> ,
  this type exchange is generally more expensive than a direct exchange (pattern matching), and only useful if you have agents which needs to listens to ALL event related to a type, but others just interested just in a few of them.

  Edit: 
  Bindings MAY be added by the sender as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1438159/+subscriptions