yahoo-eng-team team mailing list archive

[melanie witt <1866937@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We have a customer bug report downstream [1] where nova occasionally
fails to carry out server actions requiring calls to neutron API if
haproxy happens to close a connection after idle time of 10 seconds at
nearly the same time as an incoming request that attempts to re-use the
connection while it is being torn down. Here is an excerpt from [1]:

 The result of our investigation, the cause is follows.

 1. neutron-client in nova uses connection pool ( urllib3/requests ) for
http.

 2. Sometimes, http connection is reused for different requests.

 3. Connection between neutron-client and haproxy is closed from haproxy
when it is in idle for 10 seconds.

 4. If reusing connection from client side and closing connection from haproxy side are happend almost same time,
    client gets RST and end with "bad status line".

To address this problem, we can add a new config option for neutron
client (similar to the existing config options we have for cinder client
and glance client retries) to be more resilient during such scenarios.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1788853

** Affects: nova
     Importance: Low
     Assignee: melanie witt (melwitt)
         Status: New


** Tags: neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1866937

Title:
  Requests to neutron API do not use retries

Status in OpenStack Compute (nova):
  New

Bug description:
  We have a customer bug report downstream [1] where nova occasionally
  fails to carry out server actions requiring calls to neutron API if
  haproxy happens to close a connection after idle time of 10 seconds at
  nearly the same time as an incoming request that attempts to re-use
  the connection while it is being torn down. Here is an excerpt from
  [1]:

   The result of our investigation, the cause is follows.

   1. neutron-client in nova uses connection pool ( urllib3/requests )
  for http.

   2. Sometimes, http connection is reused for different requests.

   3. Connection between neutron-client and haproxy is closed from
  haproxy when it is in idle for 10 seconds.

   4. If reusing connection from client side and closing connection from haproxy side are happend almost same time,
      client gets RST and end with "bad status line".

  To address this problem, we can add a new config option for neutron
  client (similar to the existing config options we have for cinder
  client and glance client retries) to be more resilient during such
  scenarios.

  [1] https://bugzilla.redhat.com/show_bug.cgi?id=1788853

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1866937/+subscriptions