yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712412] [NEW] DVR external port setup fails with KeyError: 'host'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1712412@xxxxxxxxxxxxxxxxxx>
Date: Tue, 22 Aug 2017 19:17:16 -0000
Reply-to: Bug 1712412 <1712412@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/28/481928/9/check/gate-tempest-dsvm-neutron-
dvr-multinode-scenario-ubuntu-xenial-
nv/8fb7c92/logs/subnode-2/screen-q-l3.txt.gz?level=TRACE

Aug 22 07:55:55.697082 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 02fc503f-72c9-49bd-a4f8-30305a7efb90: KeyError: 'host'
Aug 22 07:55:55.697264 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Aug 22 07:55:55.697446 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 538, in _process_router_update
Aug 22 07:55:55.697665 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
Aug 22 07:55:55.697850 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 475, in _process_router_if_compatible
Aug 22 07:55:55.698048 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_updated_router(router)
Aug 22 07:55:55.698229 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 490, in _process_updated_router
Aug 22 07:55:55.698405 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ri.process()
Aug 22 07:55:55.698581 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 737, in process
Aug 22 07:55:55.698756 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process()
Aug 22 07:55:55.698936 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
Aug 22 07:55:55.699114 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process()
Aug 22 07:55:55.699289 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/ha_router.py", line 436, in process
Aug 22 07:55:55.699469 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
Aug 22 07:55:55.699654 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 189, in call
Aug 22 07:55:55.699903 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.logger(e)
Aug 22 07:55:55.700084 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 22 07:55:55.700260 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.force_reraise()
Aug 22 07:55:55.700457 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 22 07:55:55.700635 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
Aug 22 07:55:55.700816 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 186, in call
Aug 22 07:55:55.700995 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
Aug 22 07:55:55.701180 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1120, in process
Aug 22 07:55:55.701357 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.process_external()
Aug 22 07:55:55.701537 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 560, in process_external
Aug 22 07:55:55.701713 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external()
Aug 22 07:55:55.701886 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 895, in process_external
Aug 22 07:55:55.702070 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
Aug 22 07:55:55.702248 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 795, in _process_external_gateway
Aug 22 07:55:55.702422 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._handle_router_snat_rules(gw_port, interface_name)
Aug 22 07:55:55.702609 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_edge_router.py", line 184, in _handle_router_snat_rules
Aug 22 07:55:55.702788 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ex_gw_port, interface_name)
Aug 22 07:55:55.702980 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 514, in _handle_router_snat_rules
Aug 22 07:55:55.703155 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ext_device_name = self.get_external_device_interface_name(ex_gw_port)
Aug 22 07:55:55.703329 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 444, in get_external_device_interface_name
Aug 22 07:55:55.703503 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if not self._get_floatingips_bound_to_host(floating_ips):
Aug 22 07:55:55.703683 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 550, in _get_floatingips_bound_to_host
Aug 22 07:55:55.703856 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if (i['host'] == self.host or
Aug 22 07:55:55.704042 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent KeyError: 'host'

This is probably result of
https://review.openstack.org/#/c/437986/27/neutron/agent/l3/dvr_local_router.py
that we sqeezed late in Pike.

This is probably the root cause of ssh connectivity timeouts we observe
in dvr scenario job.

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: gate-failure l3-dvr-backlog

** Changed in: neutron
   Importance: Undecided => High

** Changed in: neutron
       Status: New => Confirmed

** Tags added: gate-failure l3-dvr-backlog

** Changed in: neutron
    Milestone: None => pike-3

** Changed in: neutron
    Milestone: pike-3 => pike-rc2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1712412

Title:
  DVR external port setup fails with KeyError: 'host'

Status in neutron:
  Confirmed

Bug description:
  http://logs.openstack.org/28/481928/9/check/gate-tempest-dsvm-neutron-
  dvr-multinode-scenario-ubuntu-xenial-
  nv/8fb7c92/logs/subnode-2/screen-q-l3.txt.gz?level=TRACE

  Aug 22 07:55:55.697082 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 02fc503f-72c9-49bd-a4f8-30305a7efb90: KeyError: 'host'
  Aug 22 07:55:55.697264 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Aug 22 07:55:55.697446 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 538, in _process_router_update
  Aug 22 07:55:55.697665 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  Aug 22 07:55:55.697850 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 475, in _process_router_if_compatible
  Aug 22 07:55:55.698048 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  Aug 22 07:55:55.698229 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 490, in _process_updated_router
  Aug 22 07:55:55.698405 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ri.process()
  Aug 22 07:55:55.698581 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 737, in process
  Aug 22 07:55:55.698756 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process()
  Aug 22 07:55:55.698936 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
  Aug 22 07:55:55.699114 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process()
  Aug 22 07:55:55.699289 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/ha_router.py", line 436, in process
  Aug 22 07:55:55.699469 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
  Aug 22 07:55:55.699654 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 189, in call
  Aug 22 07:55:55.699903 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.logger(e)
  Aug 22 07:55:55.700084 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Aug 22 07:55:55.700260 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.force_reraise()
  Aug 22 07:55:55.700457 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Aug 22 07:55:55.700635 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  Aug 22 07:55:55.700816 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 186, in call
  Aug 22 07:55:55.700995 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  Aug 22 07:55:55.701180 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1120, in process
  Aug 22 07:55:55.701357 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.process_external()
  Aug 22 07:55:55.701537 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 560, in process_external
  Aug 22 07:55:55.701713 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external()
  Aug 22 07:55:55.701886 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 895, in process_external
  Aug 22 07:55:55.702070 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  Aug 22 07:55:55.702248 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 795, in _process_external_gateway
  Aug 22 07:55:55.702422 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._handle_router_snat_rules(gw_port, interface_name)
  Aug 22 07:55:55.702609 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_edge_router.py", line 184, in _handle_router_snat_rules
  Aug 22 07:55:55.702788 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ex_gw_port, interface_name)
  Aug 22 07:55:55.702980 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 514, in _handle_router_snat_rules
  Aug 22 07:55:55.703155 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ext_device_name = self.get_external_device_interface_name(ex_gw_port)
  Aug 22 07:55:55.703329 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 444, in get_external_device_interface_name
  Aug 22 07:55:55.703503 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if not self._get_floatingips_bound_to_host(floating_ips):
  Aug 22 07:55:55.703683 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 550, in _get_floatingips_bound_to_host
  Aug 22 07:55:55.703856 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if (i['host'] == self.host or
  Aug 22 07:55:55.704042 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent KeyError: 'host'

  This is probably result of
  https://review.openstack.org/#/c/437986/27/neutron/agent/l3/dvr_local_router.py
  that we sqeezed late in Pike.

  This is probably the root cause of ssh connectivity timeouts we
  observe in dvr scenario job.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1712412/+subscriptions
Follow ups

[Bug 1712412] Re: DVR external port setup fails with KeyError: 'host'
From: OpenStack Infra, 2017-08-25