← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1565752] Re: Too many PIPEs are created when subprocess.Open fails

 

** Changed in: neutron
       Status: Confirmed => Invalid

** Changed in: neutron
     Assignee: j_king (james-agentultra) => (unassigned)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1565752

Title:
  Too many PIPEs are created when subprocess.Open fails

Status in neutron:
  Invalid

Bug description:
  1. How to reproduce:

  Set max process (soft, hard) for particular user

  Example: modify file /etc/security/limits.conf
  hunters hard    nproc           70
  hunters soft    nproc           70

  And then, start neutron-openvswitch-agent with this user. 
  Try to start many another applications to get all the free processes, then the error log will be thrown.

  In root user, check number of current open files of neutron-openvswitch-agent service.
  # ps -ef | grep neutron-openvswitch
  501      29401     1  2 Mar30 ?        03:13:53 /usr/bin/python /usr/bin/neutron-openvswitch-agent ...
  # lsof -p 29401
  neutron-o 29401 openstack   10r  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   11w  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   12r  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   13w  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   14r  FIFO        0,8      0t0 3849643464 pipe
  neutron-o 29401 openstack   15w  FIFO        0,8      0t0 3849643464 pipe
  ...

  Too many PIPE are created.

  2. Summary:

  At weekend, when server runs at high load for rotating logs or
  something else, neutron-openvswitch-agent gets error:

  2016-04-04 18:05:33.942 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception: [Errno 11] Resource temporarily unavailable
  2016-04-04 18:05:33.944 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Traceback (most recent call last):
  File "/home/hunters/neutron-7.0.0/neutron/agent/common/ovs_lib.py", line 226, in run_ofctl
  process_input=process_input)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 120, in execute
  addl_env=addl_env)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 89, in create_process
  stderr=subprocess.PIPE)
  File "/home/hunters/neutron-7.0.0/neutron/common/utils.py", line 199, in subprocess_popen
  close_fds=close_fds, env=env)
  File "/home/hunters/neutron-7.0.0/.venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in init
  subprocess_orig.Popen.init(self, args, 0, argss, *kwds)
  File "/usr/lib/python2.7/subprocess.py", line 710, in init
  errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
  self.pid = os.fork()
  OSError: [Errno 11] Resource temporarily unavailable

  And then, the PIPEs are not closed. About 700 PIPE are created. After
  2 week, it throws error "Too many open files" and then neutron-
  openvswitch-agent stops.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1565752/+subscriptions


References