yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1762886] [NEW] ha router with gateway ip create failed because of router scheduled before ha_vr_id allocated

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: shiliang <shiliang@xxxxxxxxxx>
Date: Wed, 11 Apr 2018 01:36:40 -0000
Reply-to: Bug 1762886 <1762886@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when creating a ha router with gateway ip, l3 agent throw an exception:

2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9a525821-509d-453b-b8c3-d2f192ac1beb'
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 514, in _process_router_update
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 453, in _process_router_if_compatible
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 467, in _process_updated_router
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     ri.process(self)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 518, in process
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process(agent)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 33, in process
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process(agent)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 447, in process
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(HaRouter, self).process(agent)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.logger(e)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.force_reraise()
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1153, in process
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.process_external(agent)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 472, in process_external
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external(agent)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 898, in process_external
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port, agent.pd)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 724, in _process_external_gateway
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.external_gateway_updated(ex_gw_port, interface_name)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 85, in external_gateway_updated
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     HaRouter.external_gateway_updated(self, ex_gw_port, interface_name)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 424, in external_gateway_updated
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._remove_vip(old_gateway_cidr)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 202, in _remove_vip
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     instance.remove_vip_by_ip_address(ip_cidr)
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent


and I find the reason is that a ha vr_id is not allocated when l3 agent start to instance this ha router on  the l3 agent node,that leads the ha router instance cat't get a keepalived instance through _get_keepalived_instance with ha vr_id.

** Affects: neutron
     Importance: Undecided
     Assignee: shiliang (shiliang)
         Status: New

** Attachment added: "QQ截图20180411093613.jpg"
   https://bugs.launchpad.net/bugs/1762886/+attachment/5109957/+files/QQ%E6%88%AA%E5%9B%BE20180411093613.jpg

** Changed in: networking-ml3
     Assignee: (unassigned) => shiliang (shiliang)

** Project changed: networking-ml3 => neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1762886

Title:
  ha router with gateway ip create failed because of router scheduled
  before ha_vr_id allocated

Status in neutron:
  New

Bug description:
  when creating a ha router with gateway ip, l3 agent throw an
  exception:

  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9a525821-509d-453b-b8c3-d2f192ac1beb'
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 514, in _process_router_update
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 453, in _process_router_if_compatible
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 467, in _process_updated_router
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     ri.process(self)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 518, in process
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process(agent)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 33, in process
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process(agent)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 447, in process
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(HaRouter, self).process(agent)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.logger(e)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.force_reraise()
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1153, in process
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.process_external(agent)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 472, in process_external
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external(agent)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 898, in process_external
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port, agent.pd)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 724, in _process_external_gateway
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self.external_gateway_updated(ex_gw_port, interface_name)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 85, in external_gateway_updated
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     HaRouter.external_gateway_updated(self, ex_gw_port, interface_name)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 424, in external_gateway_updated
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     self._remove_vip(old_gateway_cidr)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 202, in _remove_vip
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent     instance.remove_vip_by_ip_address(ip_cidr)
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
  2018-04-10 03:26:06.686 3667 ERROR neutron.agent.l3.agent

  
  and I find the reason is that a ha vr_id is not allocated when l3 agent start to instance this ha router on  the l3 agent node,that leads the ha router instance cat't get a keepalived instance through _get_keepalived_instance with ha vr_id.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1762886/+subscriptions
Follow ups

[Bug 1762886] [NEW] ha router with gateway ip create failed because of router scheduled before ha_vr_id allocated
From: Launchpad Bug Tracker, 2018-04-11