yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1856572] [NEW] l3 agent Bad file descriptor

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Donny Davis <donny@xxxxxxxxxxxxxx>
Date: Mon, 16 Dec 2019 14:10:37 -0000
Reply-to: Bug 1856572 <1856572@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

The latest update for Stein has started throwing this error for some
routers in the l3-agent. I have had this system in prod for about 8
months and have not had any issues until now with the l3-agent.


2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 25cd3192-d7cf-45a6-933d-4aa963900a74: OSError: [Errno 9] Bad file descriptor
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 569, in _process_router_if_compatible
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 610, in _process_updated_router
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     ri.process()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.logger(e)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.force_reraise()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1188, in process
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.process_external()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 946, in process_external
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.external_gateway_added(ex_gw_port, interface_name)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 779, in external_gateway_added
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     ex_gw_port, interface_name, self.ns_name, preserve_ips)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 728, in _external_gateway_added
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     mtu=ex_gw_port.get('mtu'))
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 265, in plug
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     namespace=namespace):
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 818, in device_exists
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return IPDevice(device_name, namespace=namespace).exists()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 318, in exists
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return privileged.interface_exists(self.name, self.namespace)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return input_func(*args, **kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent OSError: [Errno 9] Bad file descriptor
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1856572

Title:
  l3 agent Bad file descriptor

Status in neutron:
  New

Bug description:
  The latest update for Stein has started throwing this error for some
  routers in the l3-agent. I have had this system in prod for about 8
  months and have not had any issues until now with the l3-agent.



  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 25cd3192-d7cf-45a6-933d-4aa963900a74: OSError: [Errno 9] Bad file descriptor
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 569, in _process_router_if_compatible
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 610, in _process_updated_router
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     ri.process()
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.logger(e)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.force_reraise()
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1188, in process
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.process_external()
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 946, in process_external
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self.external_gateway_added(ex_gw_port, interface_name)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 779, in external_gateway_added
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     ex_gw_port, interface_name, self.ns_name, preserve_ips)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 728, in _external_gateway_added
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     mtu=ex_gw_port.get('mtu'))
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 265, in plug
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     namespace=namespace):
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 818, in device_exists
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return IPDevice(device_name, namespace=namespace).exists()
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 318, in exists
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return privileged.interface_exists(self.name, self.namespace)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return input_func(*args, **kwargs)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs)
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent OSError: [Errno 9] Bad file descriptor
  2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1856572/+subscriptions
Follow ups

[Bug 1856572] Re: l3 agent Bad file descriptor
From: Donny Davis, 2019-12-16