yahoo-eng-team team mailing list archive

[Billy Olsen <billy.olsen@xxxxxxxxxxxxx>]

I saw this last night, and can indeed confirm its related to DNS issues.
It was suggested by a colleague that it may be related to the sudo call
returning an error indicating that the hostname could not be found,
though I spent no time exploring this option today. Restarting the
openvswitch-switch service closes all the existing processes but it
seems more that the service can't stop cleanly. Also of note is that the
service stop took a very long time, whereas with working DNS it took a
few seconds at worst.

** Changed in: neutron
       Status: Expired => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1629097

Title:
  neutron-rootwrap processes not getting cleaned up

Status in neutron:
  Confirmed

Bug description:
  neutron-rootwrap processes aren't getting cleaned up on Newton.  I'm
  testing with Newton rc3.

  I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes:
  sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json

  $ top -n1 -b -o VIRT
  http://paste.ubuntu.com/23252407/

  $ ps aux|grep ovsdb-client
  http://paste.ubuntu.com/23252658/

  Restarting openvswitch cleans up the processes but they just start piling again up soon after:
  sudo systemctl restart openvswitch-switch

  At first I thought this was an openvswitch issue, however I reverted
  the code in get_root_helper_child_pid() and neutron-rootwrap processes
  started getting cleaned up. See corresponding commit for code that
  possibly introduced this at [1].

  This can be recreated with the openstack charms using xenial-newton-
  staging.  On newton deploys, neutron-gateway and nova-compute units
  will exhaust memory due to compounding ovsdb-client processes.

  [1]
  commit fd93e19f2a415b3803700fc491749daba01a4390
  Author: Assaf Muller <amuller@xxxxxxxxxx>
  Date:   Fri Mar 18 16:29:26 2016 -0400

      Change get_root_helper_child_pid to stop when it finds cmd

      get_root_helper_child_pid recursively finds the child of pid,
      until it can no longer find a child. However, the intention is
      not to find the deepest child, but to strip away root helpers.
      For example 'sudo neutron-rootwrap x' is supposed to find the
      pid of x. However, in cases 'x' spawned quick lived children of
      its own (For example: ip / brctl / ovs invocations),
      get_root_helper_child_pid returned those pids if called in
      the wrong time.

      Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
      Closes-Bug: #1558819

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1629097/+subscriptions