yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1693539] [NEW] Logstash is filled with server ERROR messages in _delete_interface_route_in_fip_ns

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1693539@xxxxxxxxxxxxxxxxxx>
Date: Thu, 25 May 2017 15:51:31 -0000
Reply-to: Bug 1693539 <1693539@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Example: http://logs.openstack.org/02/466902/8/check/gate-tempest-dsvm-
neutron-dvr-ubuntu-xenial/4fdfddd/logs/screen-q-l3.txt.gz?level=INFO (be
ware, it's huge!)

May 24 16:56:24.233922 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process
May 24 16:56:24.235058 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process
May 24 16:56:24.235237 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
May 24 16:56:24.235314 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/common/utils.py", line 183, in call
May 24 16:56:24.235388 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
May 24 16:56:24.235460 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1111, in process
May 24 16:56:24.235548 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._process_internal_ports()
May 24 16:56:24.235631 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 545, in _process_internal_ports
May 24 16:56:24.235708 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self.internal_network_removed(p)
May 24 16:56:24.235779 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 411, in internal_network_removed
May 24 16:56:24.235867 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._dvr_internal_network_removed(port)
May 24 16:56:24.235940 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_edge_router.py", line 122, in _dvr_internal_network_removed
May 24 16:56:24.236015 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     super(DvrEdgeRouter, self)._dvr_internal_network_removed(port)
May 24 16:56:24.236088 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 397, in _dvr_internal_network_removed
May 24 16:56:24.236160 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._delete_interface_route_in_fip_ns(port)
May 24 16:56:24.236237 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 561, in _delete_interface_route_in_fip_ns
May 24 16:56:24.236319 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     device.route.delete_route(rtr_port_cidr, str(rtr_2_fip_ip))
May 24 16:56:24.236400 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 816, in delete_route
May 24 16:56:24.236477 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._run_as_root_detect_device_not_found([ip_version], tuple(args))
May 24 16:56:24.236555 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 708, in _run_as_root_detect_device_not_found
May 24 16:56:24.236632 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     raise exceptions.DeviceNotFoundError(device_name=self.name)
May 24 16:56:24.236738 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
May 24 16:56:24.236815 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self.force_reraise()
May 24 16:56:24.236896 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 24 16:56:24.236969 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     six.reraise(self.type_, self.value, self.tb)
May 24 16:56:24.237046 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 703, in _run_as_root_detect_device_not_found
May 24 16:56:24.237465 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     return self._as_root(*args, **kwargs)
May 24 16:56:24.237549 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 367, in _as_root
May 24 16:56:24.237623 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
May 24 16:56:24.237704 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 100, in _as_root
May 24 16:56:24.237785 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
May 24 16:56:24.237871 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 109, in _execute
May 24 16:56:24.237957 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
May 24 16:56:24.238037 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 151, in execute
May 24 16:56:24.238218 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     raise ProcessExecutionError(msg, returncode=returncode)
May 24 16:56:24.238306 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process

** Affects: neutron
     Importance: Critical
     Assignee: Ihar Hrachyshka (ihar-hrachyshka)
         Status: In Progress


** Tags: gate-failure l3-dvr-backlog

** Changed in: neutron
   Importance: Undecided => Critical

** Changed in: neutron
       Status: New => Incomplete

** Changed in: neutron
       Status: Incomplete => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1693539

Title:
  Logstash is filled with server ERROR messages in
  _delete_interface_route_in_fip_ns

Status in neutron:
  In Progress

Bug description:
  Example: http://logs.openstack.org/02/466902/8/check/gate-tempest-
  dsvm-neutron-dvr-ubuntu-
  xenial/4fdfddd/logs/screen-q-l3.txt.gz?level=INFO (be ware, it's
  huge!)

  May 24 16:56:24.233922 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process
  May 24 16:56:24.235058 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process
  May 24 16:56:24.235237 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  May 24 16:56:24.235314 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/common/utils.py", line 183, in call
  May 24 16:56:24.235388 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  May 24 16:56:24.235460 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1111, in process
  May 24 16:56:24.235548 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._process_internal_ports()
  May 24 16:56:24.235631 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 545, in _process_internal_ports
  May 24 16:56:24.235708 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self.internal_network_removed(p)
  May 24 16:56:24.235779 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 411, in internal_network_removed
  May 24 16:56:24.235867 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._dvr_internal_network_removed(port)
  May 24 16:56:24.235940 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_edge_router.py", line 122, in _dvr_internal_network_removed
  May 24 16:56:24.236015 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     super(DvrEdgeRouter, self)._dvr_internal_network_removed(port)
  May 24 16:56:24.236088 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 397, in _dvr_internal_network_removed
  May 24 16:56:24.236160 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._delete_interface_route_in_fip_ns(port)
  May 24 16:56:24.236237 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 561, in _delete_interface_route_in_fip_ns
  May 24 16:56:24.236319 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     device.route.delete_route(rtr_port_cidr, str(rtr_2_fip_ip))
  May 24 16:56:24.236400 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 816, in delete_route
  May 24 16:56:24.236477 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self._run_as_root_detect_device_not_found([ip_version], tuple(args))
  May 24 16:56:24.236555 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 708, in _run_as_root_detect_device_not_found
  May 24 16:56:24.236632 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     raise exceptions.DeviceNotFoundError(device_name=self.name)
  May 24 16:56:24.236738 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  May 24 16:56:24.236815 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     self.force_reraise()
  May 24 16:56:24.236896 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  May 24 16:56:24.236969 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     six.reraise(self.type_, self.value, self.tb)
  May 24 16:56:24.237046 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 703, in _run_as_root_detect_device_not_found
  May 24 16:56:24.237465 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     return self._as_root(*args, **kwargs)
  May 24 16:56:24.237549 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 367, in _as_root
  May 24 16:56:24.237623 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
  May 24 16:56:24.237704 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 100, in _as_root
  May 24 16:56:24.237785 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  May 24 16:56:24.237871 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 109, in _execute
  May 24 16:56:24.237957 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
  May 24 16:56:24.238037 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 151, in execute
  May 24 16:56:24.238218 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info     raise ProcessExecutionError(msg, returncode=returncode)
  May 24 16:56:24.238306 ubuntu-xenial-osic-cloud1-s3500-8971260 neutron-l3-agent[20130]: ERROR neutron.agent.l3.router_info ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: No such process

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1693539/+subscriptions
Follow ups

[Bug 1693539] Re: Logstash is filled with server ERROR messages in _delete_interface_route_in_fip_ns
From: OpenStack Infra, 2017-05-25