yahoo-eng-team team mailing list archive

[OpenStack Infra <1581580@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/317616
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=70ea188f5d87c45fb60ace8b8405274e5f6dd489
Submitter: Jenkins
Branch:    master

commit 70ea188f5d87c45fb60ace8b8405274e5f6dd489
Author: venkata anil <anilvenkata@xxxxxxxxxx>
Date:   Tue May 17 16:30:13 2016 +0000

    New option for num_threads for state change server
    
    Currently max number of client connections(i.e greenlets spawned at
    a time) opened at any time by the WSGI server is set to 100 with
    wsgi_default_pool_size[1].
    
    This configuration may be fine for neutron api server. But with
    wsgi_default_pool_size(=100) requests, state change server
    is creating heavy cpu load on agent.
    So this server(which run on agents) need lesser value i.e
    can be configured to half the number of cpu on agent
    
    We use "ha_keepalived_state_change_server_threads" config option
    to configure number of threads in state change server instead of
    wsgi_default_pool_size.
    
    [1] https://review.openstack.org/#/c/278007/
    
    DocImpact: Add new config option -
    ha_keepalived_state_change_server_threads, to configure number
    of threads in state change server.
    
    Closes-Bug: #1581580
    Change-Id: I822ea3844792a7731fd24419b7e90e5aef141993


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1581580

Title:
  Heavy cpu load seen when keepalived state change server gets
  wsgi_default_pool_size requests at same time

Status in neutron:
  Fix Released

Bug description:
  With wsgi_default_pool_size=100[1], if the keepalived state change
  server gets 100 requests at the same time, while processing the
  requests heavy load is seen on cpu, making the network node
  unresponsive. For each request, keepalived state change server spawns
  a new meta data proxy process(i.e neutron-ns-metadata-proxy). During
  heavy cpu load, with "top" command, I can see many metadata proxy
  processes in "running" state at same time(see the attachment).

  When wsgi_default_pool_size=8, I see state change server spawning 8
  metadata proxy processes at a time("top" command shows 8 meta data
  proxy processes in "running" state at a time), cpu load is less and
  metadata proxy processes(for example, 100) spawned for all requests
  without failures(as backlog=KEEPALIVED_STATE_CHANGE_SERVER_BACKLOG i.e
  4096).

  We can keep wsgi_default_pool_size=100 for neutron API server, and use
  seperate configuration for UnixDomainWSGIServer(for example
  CONF.unix_domain_wsgi_default_pool_size).

  neutron/agent/linux/utils.py
  class UnixDomainWSGIServer(wsgi.Server):

      def _run(self, application, socket):
          """Start a WSGI service in a new green thread."""
          logger = logging.getLogger('eventlet.wsgi.server')
          eventlet.wsgi.server(socket,
                               application,
                               max_size=CONF.unix_domain_wsgi_default_pool_size,
                               protocol=UnixDomainHttpProtocol,
                               log=logger)

  [1]
  https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1581580/+subscriptions