yahoo-eng-team team mailing list archive

[Kevin Benton <1516195@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

The agent/server communication pattern we use now can lead to cascading
failures making the servers unavailable.

The current pattern in our communications between the Neutron server and
the agents looks like the following:


Server sends: item <item-uuid> changed
Client receives event.
Client makes a call to the server asking for the item details.


The calls the client makes to the server can be expensive and a server under heavy load can take a long time to start processing the request and/or to fulfill the request. This can trigger a timeout on the agent side, which leads to a retry, or, even worse, a generic fallback to resync the entire state. This creates a thundering herd problem where a server falling behind on requests will be continually stampeded by retries from agents that have timed out by the time the server can respond.


The pattern of agent/server communication needs to be adjusted to assume terrible server response times at a minimum. Optimally, all of the notifications generated by the servers should be adjusted to include all of the relevant information that an agent will need to respond to an event so the only time an agent has to actually call the server is on startup to get initial state.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1516195

Title:
  Push all object information in AMQP notifications

Status in neutron:
  New

Bug description:
  The agent/server communication pattern we use now can lead to
  cascading failures making the servers unavailable.

  The current pattern in our communications between the Neutron server
  and the agents looks like the following:

  
  Server sends: item <item-uuid> changed
  Client receives event.
  Client makes a call to the server asking for the item details.

  
  The calls the client makes to the server can be expensive and a server under heavy load can take a long time to start processing the request and/or to fulfill the request. This can trigger a timeout on the agent side, which leads to a retry, or, even worse, a generic fallback to resync the entire state. This creates a thundering herd problem where a server falling behind on requests will be continually stampeded by retries from agents that have timed out by the time the server can respond.

  
  The pattern of agent/server communication needs to be adjusted to assume terrible server response times at a minimum. Optimally, all of the notifications generated by the servers should be adjusted to include all of the relevant information that an agent will need to respond to an event so the only time an agent has to actually call the server is on startup to get initial state.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1516195/+subscriptions