← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #83658

[Bug 1892846] [NEW] AttributeError in l3-agent changing HA router to non-HA

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

In case when router is migrated from HA to non-HA one, it has to be disabled and then migrated.
If that is done fast enough, and some error will occur in router_remove method (https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L547) old router_info will be added to the self.router_info dict.
In such case it will be still HA router added to self.router_info but as this router is already switched to be non-HA, it will use wrong RouterInfo class and errors like:

2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'remove_vip_by_ip_address': AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._process_internal_ports()
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self.internal_network_added(p)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._disable_ipv6_addressing_on_interface(interface_name)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._remove_vip(ipv6_lladdr)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     instance.remove_vip_by_ip_address(ip_cidr)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 8a8e034d-0c53-4d06-a936-9bb53b064f02: AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     ri.process()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 485, in process
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.logger(e)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     raise value
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_internal_ports()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.internal_network_added(p)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._disable_ipv6_addressing_on_interface(interface_name)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._remove_vip(ipv6_lladdr)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     instance.remove_vip_by_ip_address(ip_cidr)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'

will occur in L3 agent.

** Affects: neutron
     Importance: Medium
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: l3-ha

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1892846

Title:
  AttributeError in l3-agent changing HA router to non-HA

Status in neutron:
  Confirmed

Bug description:
  In case when router is migrated from HA to non-HA one, it has to be disabled and then migrated.
  If that is done fast enough, and some error will occur in router_remove method (https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L547) old router_info will be added to the self.router_info dict.
  In such case it will be still HA router added to self.router_info but as this router is already switched to be non-HA, it will use wrong RouterInfo class and errors like:

  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'remove_vip_by_ip_address': AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._process_internal_ports()
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self.internal_network_added(p)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._disable_ipv6_addressing_on_interface(interface_name)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     self._remove_vip(ipv6_lladdr)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info     instance.remove_vip_by_ip_address(ip_cidr)
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
  2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 8a8e034d-0c53-4d06-a936-9bb53b064f02: AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     ri.process()
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 485, in process
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.logger(e)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     raise value
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._process_internal_ports()
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self.internal_network_added(p)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._disable_ipv6_addressing_on_interface(interface_name)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     self._remove_vip(ipv6_lladdr)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent     instance.remove_vip_by_ip_address(ip_cidr)
  2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'

  will occur in L3 agent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1892846/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next