yahoo-eng-team team mailing list archive

[venkata anil <anilvenkata@xxxxxxxxxx>]

Public bug reported:

With wsgi_default_pool_size=100[1], if the keepalived state change
server gets 100 requests at the same time, while processing the requests
heavy load is seen on cpu, making the network node unresponsive. For
each request, keepalived state change server spawns a new meta data
proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load,
with "top" command, I can see many metadata proxy processes in "running"
state at same time(see the attachment).

When wsgi_default_pool_size=8, I see state change server spawning 8
metadata proxy processes at a time("top" command shows 8 meta data proxy
processes in "running" state at a time), cpu load is less and metadata
proxy processes(for example, 100) spawned for all requests without
failures.

We can keep wsgi_default_pool_size=100 for neutron API server, and use
seperate configuration for UnixDomainWSGIServer(for example
CONF.unix_domain_wsgi_default_pool_size).

neutron/agent/linux/utils.py
class UnixDomainWSGIServer(wsgi.Server):

    def _run(self, application, socket):
        """Start a WSGI service in a new green thread."""
        logger = logging.getLogger('eventlet.wsgi.server') 
        eventlet.wsgi.server(socket, 
                             application,
                             max_size=CONF.unix_domain_wsgi_default_pool_size,
                             protocol=UnixDomainHttpProtocol,
                             log=logger)

[1]
https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807

** Affects: neutron
     Importance: Undecided
     Assignee: venkata anil (anil-venkata)
         Status: New


** Tags: l3-ha

** Attachment added: "top.gif"
   https://bugs.launchpad.net/bugs/1581580/+attachment/4662237/+files/top.gif

** Changed in: neutron
     Assignee: (unassigned) => venkata anil (anil-venkata)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1581580

Title:
  Heavy cpu load seen when keepalived state change server gets
  wsi_default_pool_size requests at same time

Status in neutron:
  New

Bug description:
  With wsgi_default_pool_size=100[1], if the keepalived state change
  server gets 100 requests at the same time, while processing the
  requests heavy load is seen on cpu, making the network node
  unresponsive. For each request, keepalived state change server spawns
  a new meta data proxy process(i.e neutron-ns-metadata-proxy). During
  heavy cpu load, with "top" command, I can see many metadata proxy
  processes in "running" state at same time(see the attachment).

  When wsgi_default_pool_size=8, I see state change server spawning 8
  metadata proxy processes at a time("top" command shows 8 meta data
  proxy processes in "running" state at a time), cpu load is less and
  metadata proxy processes(for example, 100) spawned for all requests
  without failures.

  We can keep wsgi_default_pool_size=100 for neutron API server, and use
  seperate configuration for UnixDomainWSGIServer(for example
  CONF.unix_domain_wsgi_default_pool_size).

  neutron/agent/linux/utils.py
  class UnixDomainWSGIServer(wsgi.Server):

      def _run(self, application, socket):
          """Start a WSGI service in a new green thread."""
          logger = logging.getLogger('eventlet.wsgi.server') 
          eventlet.wsgi.server(socket, 
                               application,
                               max_size=CONF.unix_domain_wsgi_default_pool_size,
                               protocol=UnixDomainHttpProtocol,
                               log=logger)

  [1]
  https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1581580/+subscriptions