yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1817435] [NEW] Error while stoping radvd on dvr ha router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1817435@xxxxxxxxxxxxxxxxxx>
Date: Sat, 23 Feb 2019 21:42:37 -0000
Reply-to: Bug 1817435 <1817435@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In neutron-tempest-dvr-multinode-ha-full I found errors in L3 agent's logs. Error example is on:
http://logs.openstack.org/79/633979/11/check/neutron-tempest-dvr-ha-multinode-full/415f056/controller/logs/screen-q-l3.txt.gz?level=ERROR#_Feb_22_12_00_34_232612

Error looks like:

Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Error while deleting router 1f90a750-d3af-454a-9031-29f399cf6457: AttributeError: 'NoneType' object has no attribute 'disable'
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 447, in _router_added
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     ri.delete()
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 461, in delete
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     super(HaRouter, self).delete()
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 431, in delete
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.disable_radvd()
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 547, in disable_radvd
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.radvd.disable()
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'disable'
Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 1f90a750-d3af-454a-9031-29f399cf6457: Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 687, in _process_routers_if_compatible
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 562, in _process_router_if_compatible
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._process_added_router(router)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 567, in _process_added_router
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 450, in _router_added
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     router_id)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.force_reraise()
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 439, in _router_added
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     ri.initialize(self.process_monitor)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 126, in initialize
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     raise Exception(msg)
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 


Funny thing is that I found it on my patch https://review.openstack.org/#/c/633979/ which is migrating job definition to zuulv3 syntax. There is no such error in existing legacy job because it is not creates HA routers by default. So this is another issue that neutron-tempest-dvr-HA-multinode-full job is actually not HA but that can be fixed in https://review.openstack.org/#/c/633979/ after this issue reported here will be fixed.

** Affects: neutron
     Importance: Medium
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: gate-failure l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1817435

Title:
  Error while stoping radvd on dvr ha router

Status in neutron:
  Confirmed

Bug description:
  In neutron-tempest-dvr-multinode-ha-full I found errors in L3 agent's logs. Error example is on:
  http://logs.openstack.org/79/633979/11/check/neutron-tempest-dvr-ha-multinode-full/415f056/controller/logs/screen-q-l3.txt.gz?level=ERROR#_Feb_22_12_00_34_232612

  Error looks like:

  Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
  Feb 22 12:00:34.219810 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Error while deleting router 1f90a750-d3af-454a-9031-29f399cf6457: AttributeError: 'NoneType' object has no attribute 'disable'
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 447, in _router_added
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     ri.delete()
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 461, in delete
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     super(HaRouter, self).delete()
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 431, in delete
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.disable_radvd()
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 547, in disable_radvd
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.radvd.disable()
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'disable'
  Feb 22 12:00:34.232612 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 1f90a750-d3af-454a-9031-29f399cf6457: Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 687, in _process_routers_if_compatible
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 562, in _process_router_if_compatible
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._process_added_router(router)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 567, in _process_added_router
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 450, in _router_added
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     router_id)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     self.force_reraise()
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 439, in _router_added
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     ri.initialize(self.process_monitor)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 126, in initialize
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent     raise Exception(msg)
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent Exception: Unable to process HA router 1f90a750-d3af-454a-9031-29f399cf6457 without HA port
  Feb 22 12:00:34.235509 ubuntu-bionic-rax-ord-0002911263 neutron-l3-agent[21802]: ERROR neutron.agent.l3.agent 

  
  Funny thing is that I found it on my patch https://review.openstack.org/#/c/633979/ which is migrating job definition to zuulv3 syntax. There is no such error in existing legacy job because it is not creates HA routers by default. So this is another issue that neutron-tempest-dvr-HA-multinode-full job is actually not HA but that can be fixed in https://review.openstack.org/#/c/633979/ after this issue reported here will be fixed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1817435/+subscriptions
Follow ups

[Bug 1817435] Re: Error while stoping radvd on dvr ha router
From: OpenStack Infra, 2019-03-02