← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #48925

[Bug 1565752] [NEW] Too many PIPEs are created when subprocess.Open fails

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

1. How to reproduce:

Set max process (soft, hard) for particular user

Example: modify file /etc/security/limits.conf
hunters hard    nproc           70
hunters soft    nproc           70

And then, start neutron-openvswitch-agent with this user. 
Try to start many another applications to get all the free processes, then the error log will be thrown.

In root user, check number of current open files of neutron-openvswitch-agent service.
# ps -ef | grep neutron-openvswitch
501      29401     1  2 Mar30 ?        03:13:53 /usr/bin/python /usr/bin/neutron-openvswitch-agent ...
# lsof -p 29401
neutron-o 29401 openstack   10r  FIFO        0,8      0t0 3849643462 pipe
neutron-o 29401 openstack   11w  FIFO        0,8      0t0 3849643462 pipe
neutron-o 29401 openstack   12r  FIFO        0,8      0t0 3849643463 pipe
neutron-o 29401 openstack   13w  FIFO        0,8      0t0 3849643463 pipe
neutron-o 29401 openstack   14r  FIFO        0,8      0t0 3849643464 pipe
neutron-o 29401 openstack   15w  FIFO        0,8      0t0 3849643464 pipe
...

Too many PIPE are created.

2. Summary:

At weekend, when server runs at high load for rotating logs or something
else, neutron-openvswitch-agent gets error:

2016-04-04 18:05:33.942 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception: [Errno 11] Resource temporarily unavailable
2016-04-04 18:05:33.944 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Traceback (most recent call last):
File "/home/hunters/neutron-7.0.0/neutron/agent/common/ovs_lib.py", line 226, in run_ofctl
process_input=process_input)
File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 120, in execute
addl_env=addl_env)
File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 89, in create_process
stderr=subprocess.PIPE)
File "/home/hunters/neutron-7.0.0/neutron/common/utils.py", line 199, in subprocess_popen
close_fds=close_fds, env=env)
File "/home/hunters/neutron-7.0.0/.venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in init
subprocess_orig.Popen.init(self, args, 0, argss, *kwds)
File "/usr/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

And then, the PIPEs are not closed. About 700 PIPE are created. After 2
week, it throws error "Too many open files" and then neutron-
openvswitch-agent stops.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

- How to reproduce:
+ 1. How to reproduce:
  
  Set max process (soft, hard) for particular user
  
  Example: modify file /etc/security/limits.conf
  hunters hard    nproc           70
  hunters soft    nproc           70
  
  And then, start neutron-openvswitch-agent with this user.
  
  In root user, check number of current open files of neutron-openvswitch-agent service.
  # ps -ef | grep neutron-openvswitch
  501      29401     1  2 Mar30 ?        03:13:53 /usr/bin/python /usr/bin/neutron-openvswitch-agent ...
  # lsof -p 29401
  neutron-o 29401 openstack   10r  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   11w  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   12r  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   13w  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   14r  FIFO        0,8      0t0 3849643464 pipe
  neutron-o 29401 openstack   15w  FIFO        0,8      0t0 3849643464 pipe
  ...
  
  Too many PIPE are created.
+ 
+ 2. Summary:
  
  At weekend, when server runs at high load for rotating logs or something
  else, neutron-openvswitch-agent gets error:
  
  2016-04-04 18:05:33.942 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception: [Errno 11] Resource temporarily unavailable
  2016-04-04 18:05:33.944 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Traceback (most recent call last):
  File "/home/hunters/neutron-7.0.0/neutron/agent/common/ovs_lib.py", line 226, in run_ofctl
  process_input=process_input)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 120, in execute
  addl_env=addl_env)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 89, in create_process
  stderr=subprocess.PIPE)
  File "/home/hunters/neutron-7.0.0/neutron/common/utils.py", line 199, in subprocess_popen
  close_fds=close_fds, env=env)
  File "/home/hunters/neutron-7.0.0/.venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in init
  subprocess_orig.Popen.init(self, args, 0, argss, *kwds)
  File "/usr/lib/python2.7/subprocess.py", line 710, in init
  errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
  self.pid = os.fork()
  OSError: [Errno 11] Resource temporarily unavailable
  
  And then, the PIPEs are not closed. About 700 PIPE are created. After 2
  week, it throws error "Too many open files" and then neutron-
  openvswitch-agent stops.

** Description changed:

  1. How to reproduce:
  
  Set max process (soft, hard) for particular user
  
  Example: modify file /etc/security/limits.conf
  hunters hard    nproc           70
  hunters soft    nproc           70
  
- And then, start neutron-openvswitch-agent with this user.
+ And then, start neutron-openvswitch-agent with this user. 
+ Try to start many another applications to get all the free processes, then the error log will be thrown.
  
  In root user, check number of current open files of neutron-openvswitch-agent service.
  # ps -ef | grep neutron-openvswitch
  501      29401     1  2 Mar30 ?        03:13:53 /usr/bin/python /usr/bin/neutron-openvswitch-agent ...
  # lsof -p 29401
  neutron-o 29401 openstack   10r  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   11w  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   12r  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   13w  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   14r  FIFO        0,8      0t0 3849643464 pipe
  neutron-o 29401 openstack   15w  FIFO        0,8      0t0 3849643464 pipe
  ...
  
  Too many PIPE are created.
  
  2. Summary:
  
  At weekend, when server runs at high load for rotating logs or something
  else, neutron-openvswitch-agent gets error:
  
  2016-04-04 18:05:33.942 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception: [Errno 11] Resource temporarily unavailable
  2016-04-04 18:05:33.944 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Traceback (most recent call last):
  File "/home/hunters/neutron-7.0.0/neutron/agent/common/ovs_lib.py", line 226, in run_ofctl
  process_input=process_input)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 120, in execute
  addl_env=addl_env)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 89, in create_process
  stderr=subprocess.PIPE)
  File "/home/hunters/neutron-7.0.0/neutron/common/utils.py", line 199, in subprocess_popen
  close_fds=close_fds, env=env)
  File "/home/hunters/neutron-7.0.0/.venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in init
  subprocess_orig.Popen.init(self, args, 0, argss, *kwds)
  File "/usr/lib/python2.7/subprocess.py", line 710, in init
  errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
  self.pid = os.fork()
  OSError: [Errno 11] Resource temporarily unavailable
  
  And then, the PIPEs are not closed. About 700 PIPE are created. After 2
  week, it throws error "Too many open files" and then neutron-
  openvswitch-agent stops.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1565752

Title:
  Too many PIPEs are created when subprocess.Open fails

Status in neutron:
  New

Bug description:
  1. How to reproduce:

  Set max process (soft, hard) for particular user

  Example: modify file /etc/security/limits.conf
  hunters hard    nproc           70
  hunters soft    nproc           70

  And then, start neutron-openvswitch-agent with this user. 
  Try to start many another applications to get all the free processes, then the error log will be thrown.

  In root user, check number of current open files of neutron-openvswitch-agent service.
  # ps -ef | grep neutron-openvswitch
  501      29401     1  2 Mar30 ?        03:13:53 /usr/bin/python /usr/bin/neutron-openvswitch-agent ...
  # lsof -p 29401
  neutron-o 29401 openstack   10r  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   11w  FIFO        0,8      0t0 3849643462 pipe
  neutron-o 29401 openstack   12r  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   13w  FIFO        0,8      0t0 3849643463 pipe
  neutron-o 29401 openstack   14r  FIFO        0,8      0t0 3849643464 pipe
  neutron-o 29401 openstack   15w  FIFO        0,8      0t0 3849643464 pipe
  ...

  Too many PIPE are created.

  2. Summary:

  At weekend, when server runs at high load for rotating logs or
  something else, neutron-openvswitch-agent gets error:

  2016-04-04 18:05:33.942 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Unable to execute ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23']. Exception: [Errno 11] Resource temporarily unavailable
  2016-04-04 18:05:33.944 7817 ERROR neutron.agent.common.ovs_lib [req-42b082b1-2fbf-48a2-b2f3-3b7d774141f0 - - - - -] Traceback (most recent call last):
  File "/home/hunters/neutron-7.0.0/neutron/agent/common/ovs_lib.py", line 226, in run_ofctl
  process_input=process_input)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 120, in execute
  addl_env=addl_env)
  File "/home/hunters/neutron-7.0.0/neutron/agent/linux/utils.py", line 89, in create_process
  stderr=subprocess.PIPE)
  File "/home/hunters/neutron-7.0.0/neutron/common/utils.py", line 199, in subprocess_popen
  close_fds=close_fds, env=env)
  File "/home/hunters/neutron-7.0.0/.venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in init
  subprocess_orig.Popen.init(self, args, 0, argss, *kwds)
  File "/usr/lib/python2.7/subprocess.py", line 710, in init
  errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
  self.pid = os.fork()
  OSError: [Errno 11] Resource temporarily unavailable

  And then, the PIPEs are not closed. About 700 PIPE are created. After
  2 week, it throws error "Too many open files" and then neutron-
  openvswitch-agent stops.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1565752/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next