yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1510757] Re: tempest test api.network; l3 agent can't delete HA-router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alejandro Comisario <alejandro@xxxxxxxxxxx>
Date: Mon, 03 Apr 2017 17:05:46 -0000
Reply-to: Bug 1510757 <1510757@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Happening on mitaka ubuntu packages as of today.
And also, when you create the router, you also get on l3-agent :

2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'config'
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 379, in call
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1002, in process
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     self.process_external(agent)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 515, in process_external
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     super(DvrLocalRouter, self).process_external(agent)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 827, in process_external
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port, agent.pd)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 701, in _process_external_gateway
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     self.external_gateway_added(ex_gw_port, interface_name)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 61, in external_gateway_added
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     ex_gw_port, interface_name)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_router.py", line 45, in external_gateway_added
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     self.routes_updated([], self.router['routes'])
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 203, in routes_updated
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     instance = self._get_keepalived_instance()
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 150, in _get_keepalived_instance
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info     return self.keepalived_manager.config.get_instance(self.ha_vr_id)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'config'
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '5875dcd3-58d7-4e11-8d3a-3c15606a12ef'
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self._process_added_router(router)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     ri.process(self)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 578, in process
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process(agent)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process(agent)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 403, in process
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     super(HaRouter, self).process(agent)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 382, in call
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self.logger(e)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self.force_reraise()
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 379, in call
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1002, in process
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self.process_external(agent)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 515, in process_external
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external(agent)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 827, in process_external
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port, agent.pd)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 701, in _process_external_gateway
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self.external_gateway_added(ex_gw_port, interface_name)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 61, in external_gateway_added
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     ex_gw_port, interface_name)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_router.py", line 45, in external_gateway_added
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     self.routes_updated([], self.router['routes'])
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 203, in routes_updated
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     instance = self._get_keepalived_instance()
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 150, in _get_keepalived_instance
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent     return self.keepalived_manager.config.get_instance(self.ha_vr_id)
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'config'
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent

** Changed in: neutron
       Status: Expired => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1510757

Title:
  tempest test api.network;l3 agent can't delete HA-router

Status in neutron:
  Confirmed

Bug description:
  I use tempest test my company's openstack environment.

  # tox -eall -- tempest.api.network

  When tox finished. Log in L3-agent always show:

  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent [-] Error while deleting router da4b28ce-33b1-4000-8609-a41a2ab8c982
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 344, in _safe_router_removed
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 362, in _router_removed
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     ri.delete(self)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 364, in delete
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     super(HaRouter, self).delete(agent)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 273, in delete
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     self.process(agent)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 370, in process
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     super(HaRouter, self).process(agent)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 359, in call
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     self.logger(e)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 356, in call
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 695, in process
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     self.routes_updated()
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 181, in routes_updated
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     instance = self._get_keepalived_instance()
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 131, in _get_keepalived_instance
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent     return self.keepalived_manager.config.get_instance(self.ha_vr_id)
  2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'config'

  I think the reason is tempest create and delete router too fast.
  when l3-agent create ha-router,tempest delete the router,and neutron-server delete ha-interface

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L79

  class HaRouter can't initialized without ha-interface information,it just return without _init_keepalived_manager
  When l3-agent delete router,it report AttributeError: 'NoneType'.

  When l3 agent can't delete the router,l3 agent always fullsync with neutron-server every 30 seconds.
  In controller, the neutron-server cpu always 70%..^-^

  L3 agent should add a check before create HArouter.If it find ha-
  interface is none ,means router has been deleted in neutron-server.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1510757/+subscriptions
References

[Bug 1510757] [NEW] tempest test api.network; l3 agent can't delete HA-router
From: zhang sheng, 2015-10-28