yahoo-eng-team team mailing list archive

["Le, Huifeng" <1800599@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

High level description:
The current openstack API server uses eventlet.wsgi.server implementation. The default behavior of eventlet.wsgi.server will do an accept() call before knowing whether a greenthread is available in the pool to service that socket. If all socket connections are shortlived then this is not an issue as a greenthread will eventually become available and the request will be serviced (hopefully before the client times out waiting).

But in some scenarios in real system, such as during large system
deployment stage, there are many compute nodes which caused many long-
lived connections from nova-compute to the neutron API, this will cause
issue/unexpected behavior as below:

1. for single neutron server case:
if neutron server has all of its greenthreads tied up on open sockets, when one more connection request arrives, the server call accept() but will never distribute it to a working thread to process it and the client will timeout with long time waiting (e.g. CONF.client_socket_timeout)

Expect behavior: return quick TCP connect timeout if no processing
thread available

2. for multiple neutron server cases (e.g. cfg.CONF.api_workers>1 or cpu_count>1):
in this case, there are multiple neutron server child processes waiting for client requests (e.g. doing accept() on the same socket), if one neutron server's accept() is invoked by linux kernel to accept a client request but all of its greenthreads had tied up on open sockets then the client will timeout with long time waiting. But actually, at this time, other neutron child processes may still have available greenthreads to process this request but there is no opportunity for them to process it (as accepted by the first neutron server child process).

Expect behavior: the request can be processed if any of neutron server
process has available greenthread or return quick TCP connect timeout if
no processing thread available

Version: latest devstack

Potential solution: implement a custom pool for wsgi.server which will
block the spawn_n call (e.g. by sem.acquire()) to avoid calling accept()
until green working thread available.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1800599

Title:
  Neutron API server: unexpected behavior with multiple long live
  clients

Status in neutron:
  New

Bug description:
  High level description:
  The current openstack API server uses eventlet.wsgi.server implementation. The default behavior of eventlet.wsgi.server will do an accept() call before knowing whether a greenthread is available in the pool to service that socket. If all socket connections are shortlived then this is not an issue as a greenthread will eventually become available and the request will be serviced (hopefully before the client times out waiting).

  But in some scenarios in real system, such as during large system
  deployment stage, there are many compute nodes which caused many long-
  lived connections from nova-compute to the neutron API, this will
  cause issue/unexpected behavior as below:

  1. for single neutron server case:
  if neutron server has all of its greenthreads tied up on open sockets, when one more connection request arrives, the server call accept() but will never distribute it to a working thread to process it and the client will timeout with long time waiting (e.g. CONF.client_socket_timeout)

  Expect behavior: return quick TCP connect timeout if no processing
  thread available

  2. for multiple neutron server cases (e.g. cfg.CONF.api_workers>1 or cpu_count>1):
  in this case, there are multiple neutron server child processes waiting for client requests (e.g. doing accept() on the same socket), if one neutron server's accept() is invoked by linux kernel to accept a client request but all of its greenthreads had tied up on open sockets then the client will timeout with long time waiting. But actually, at this time, other neutron child processes may still have available greenthreads to process this request but there is no opportunity for them to process it (as accepted by the first neutron server child process).

  Expect behavior: the request can be processed if any of neutron server
  process has available greenthread or return quick TCP connect timeout
  if no processing thread available

  Version: latest devstack

  Potential solution: implement a custom pool for wsgi.server which will
  block the spawn_n call (e.g. by sem.acquire()) to avoid calling
  accept() until green working thread available.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1800599/+subscriptions