yahoo-eng-team team mailing list archive

[James Page <james.page@xxxxxxxxxx>]

Public bug reported:

OpenStack Juno, Ubuntu 14.04

500 compute node cloud, running 4.5k active instances (can't get it any
further right now).

As the number of instances in the cloud increases, the idle loading on
the neutron-server servers (4 of them all with 4 cores/8 threads and a
suitable *_worker configuration) increases from nothing to 30;  The RPC
call get_port_and_sgs is being serviced around 10 times per second on
each server at this point.

The result is that its no longer possible to create new instances, as
the rpc calls and api thread just don't get onto CPU, resulting in VIF
plugging timeouts on compute nodes, and ERROR'ed instances.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml";,
             }
Date: Thu Oct 23 10:22:14 2014
PackageArchitecture: all
SourcePackage: neutron
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.neutron.api.paste.ini: [deleted]
modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
modified.conffile..etc.neutron.l3.agent.ini: [deleted]
modified.conffile..etc.neutron.neutron.conf: [deleted]
modified.conffile..etc.neutron.policy.json: [deleted]
modified.conffile..etc.neutron.rootwrap.conf: [deleted]
modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

** Affects: cloud-archive
     Importance: Undecided
         Status: New

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug juno scale-test third-party-packages trusty uec-images

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1384660

Title:
  Idle rpc traffic with a large number of instances causes failures

Status in Ubuntu Cloud Archive:
  New
Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  OpenStack Juno, Ubuntu 14.04

  500 compute node cloud, running 4.5k active instances (can't get it
  any further right now).

  As the number of instances in the cloud increases, the idle loading on
  the neutron-server servers (4 of them all with 4 cores/8 threads and a
  suitable *_worker configuration) increases from nothing to 30;  The
  RPC call get_port_and_sgs is being serviced around 10 times per second
  on each server at this point.

  The result is that its no longer possible to create new instances, as
  the rpc calls and api thread just don't get onto CPU, resulting in VIF
  plugging timeouts on compute nodes, and ERROR'ed instances.

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
  ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
  Uname: Linux 3.13.0-35-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml";,
               }
  Date: Thu Oct 23 10:22:14 2014
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.neutron.api.paste.ini: [deleted]
  modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
  modified.conffile..etc.neutron.l3.agent.ini: [deleted]
  modified.conffile..etc.neutron.neutron.conf: [deleted]
  modified.conffile..etc.neutron.policy.json: [deleted]
  modified.conffile..etc.neutron.rootwrap.conf: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
  modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
  modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1384660/+subscriptions