yahoo-eng-team team mailing list archive

[Corey Bryant <corey.bryant@xxxxxxxxxxxxx>]

** Also affects: neutron
   Importance: Undecided
       Status: New

** No longer affects: openvswitch (Ubuntu)

** Summary changed:

- ovsdb-client processes not getting cleaned up
+ neutron-rootwrap processes not getting cleaned up

** Description changed:

- This can be recreated with the openstack charms using xenial-newton-staging.  On newton deploys, neutron-gateway and nova-compute units will exhaust memory due to compounding ovsdb-client processes:
+ neutron-rootwrap processes aren't getting cleaned up on Newton.  I'm
+ testing with Newton rc3.
+ 
+ I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes:
  sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
  
  $ top -n1 -b -o VIRT
  http://paste.ubuntu.com/23252407/
  
  $ ps aux|grep ovsdb-client
  http://paste.ubuntu.com/23252658/
  
  Restarting openvswitch cleans up the processes but they just start piling again up soon after:
  sudo systemctl restart openvswitch-switch
+ 
+ At first I thought this was an openvswitch issue, however I reverted the
+ code in get_root_helper_child_pid() and neutron-rootwrap processes
+ started getting cleaned up. See corresponding commit at [1].
+ 
+ This can be recreated with the openstack charms using xenial-newton-
+ staging.  On newton deploys, neutron-gateway and nova-compute units will
+ exhaust memory due to compounding ovsdb-client processes.
+ 
+ [1]
+ commit fd93e19f2a415b3803700fc491749daba01a4390
+ Author: Assaf Muller <amuller@xxxxxxxxxx>
+ Date:   Fri Mar 18 16:29:26 2016 -0400
+ 
+     Change get_root_helper_child_pid to stop when it finds cmd
+     
+     get_root_helper_child_pid recursively finds the child of pid,
+     until it can no longer find a child. However, the intention is
+     not to find the deepest child, but to strip away root helpers.
+     For example 'sudo neutron-rootwrap x' is supposed to find the
+     pid of x. However, in cases 'x' spawned quick lived children of
+     its own (For example: ip / brctl / ovs invocations),
+     get_root_helper_child_pid returned those pids if called in
+     the wrong time.
+     
+     Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
+     Closes-Bug: #1558819

** Description changed:

  neutron-rootwrap processes aren't getting cleaned up on Newton.  I'm
  testing with Newton rc3.
  
  I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes:
  sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
  
  $ top -n1 -b -o VIRT
  http://paste.ubuntu.com/23252407/
  
  $ ps aux|grep ovsdb-client
  http://paste.ubuntu.com/23252658/
  
  Restarting openvswitch cleans up the processes but they just start piling again up soon after:
  sudo systemctl restart openvswitch-switch
  
  At first I thought this was an openvswitch issue, however I reverted the
  code in get_root_helper_child_pid() and neutron-rootwrap processes
- started getting cleaned up. See corresponding commit at [1].
+ started getting cleaned up. See corresponding commit for code that
+ possibly introduced this at [1].
  
  This can be recreated with the openstack charms using xenial-newton-
  staging.  On newton deploys, neutron-gateway and nova-compute units will
  exhaust memory due to compounding ovsdb-client processes.
  
  [1]
  commit fd93e19f2a415b3803700fc491749daba01a4390
  Author: Assaf Muller <amuller@xxxxxxxxxx>
  Date:   Fri Mar 18 16:29:26 2016 -0400
  
-     Change get_root_helper_child_pid to stop when it finds cmd
-     
-     get_root_helper_child_pid recursively finds the child of pid,
-     until it can no longer find a child. However, the intention is
-     not to find the deepest child, but to strip away root helpers.
-     For example 'sudo neutron-rootwrap x' is supposed to find the
-     pid of x. However, in cases 'x' spawned quick lived children of
-     its own (For example: ip / brctl / ovs invocations),
-     get_root_helper_child_pid returned those pids if called in
-     the wrong time.
-     
-     Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
-     Closes-Bug: #1558819
+     Change get_root_helper_child_pid to stop when it finds cmd
+ 
+     get_root_helper_child_pid recursively finds the child of pid,
+     until it can no longer find a child. However, the intention is
+     not to find the deepest child, but to strip away root helpers.
+     For example 'sudo neutron-rootwrap x' is supposed to find the
+     pid of x. However, in cases 'x' spawned quick lived children of
+     its own (For example: ip / brctl / ovs invocations),
+     get_root_helper_child_pid returned those pids if called in
+     the wrong time.
+ 
+     Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
+     Closes-Bug: #1558819

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1629097

Title:
  neutron-rootwrap processes not getting cleaned up

Status in neutron:
  New

Bug description:
  neutron-rootwrap processes aren't getting cleaned up on Newton.  I'm
  testing with Newton rc3.

  I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes:
  sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json

  $ top -n1 -b -o VIRT
  http://paste.ubuntu.com/23252407/

  $ ps aux|grep ovsdb-client
  http://paste.ubuntu.com/23252658/

  Restarting openvswitch cleans up the processes but they just start piling again up soon after:
  sudo systemctl restart openvswitch-switch

  At first I thought this was an openvswitch issue, however I reverted
  the code in get_root_helper_child_pid() and neutron-rootwrap processes
  started getting cleaned up. See corresponding commit for code that
  possibly introduced this at [1].

  This can be recreated with the openstack charms using xenial-newton-
  staging.  On newton deploys, neutron-gateway and nova-compute units
  will exhaust memory due to compounding ovsdb-client processes.

  [1]
  commit fd93e19f2a415b3803700fc491749daba01a4390
  Author: Assaf Muller <amuller@xxxxxxxxxx>
  Date:   Fri Mar 18 16:29:26 2016 -0400

      Change get_root_helper_child_pid to stop when it finds cmd

      get_root_helper_child_pid recursively finds the child of pid,
      until it can no longer find a child. However, the intention is
      not to find the deepest child, but to strip away root helpers.
      For example 'sudo neutron-rootwrap x' is supposed to find the
      pid of x. However, in cases 'x' spawned quick lived children of
      its own (For example: ip / brctl / ovs invocations),
      get_root_helper_child_pid returned those pids if called in
      the wrong time.

      Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
      Closes-Bug: #1558819

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1629097/+subscriptions