yahoo-eng-team team mailing list archive

[Rodolfo Alonso <1384660@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: Incomplete => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1384660

Title:
  Idle rpc traffic with a large number of instances causes failures

Status in Ubuntu Cloud Archive:
  Fix Released
Status in neutron:
  Fix Released

Bug description:
  OpenStack Juno (Neutron ML2+OVS/l2pop/neutron security groups), Ubuntu
  14.04

  500 compute node cloud, running 4.5k active instances (can't get it
  any further right now).

  As the number of instances in the cloud increases, the idle loading on
  the neutron-server servers (4 of them all with 4 cores/8 threads and a
  suitable *_worker configuration) increases from nothing to 30;  The db
  call get_port_and_sgs is being serviced around 10 times per second on
  each server at this point. Other things are also happening - I've
  attached the last 1000 lines of the server log with debug enabled.

  The result is that its no longer possible to create new instances, as
  the rpc calls and api thread just don't get onto CPU, resulting in VIF
  plugging timeouts on compute nodes, and ERROR'ed instances.

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
  ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
  Uname: Linux 3.13.0-35-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml";,
               }
  Date: Thu Oct 23 10:22:14 2014
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.neutron.api.paste.ini: [deleted]
  modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
  modified.conffile..etc.neutron.l3.agent.ini: [deleted]
  modified.conffile..etc.neutron.neutron.conf: [deleted]
  modified.conffile..etc.neutron.policy.json: [deleted]
  modified.conffile..etc.neutron.rootwrap.conf: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
  modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
  modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1384660/+subscriptions