yahoo-eng-team team mailing list archive

[OpenStack Infra <1712412@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/497009
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=47fbc6157ac4125fa068e64b573433a02c0ce0fc
Submitter: Jenkins
Branch:    master

commit 47fbc6157ac4125fa068e64b573433a02c0ce0fc
Author: Swaminathan Vasudevan <SVasudevan@xxxxxxxx>
Date:   Wed Aug 23 21:11:58 2017 -0700

    DVR: _get_floatingips_bound_to_host throws KeyError
    
    _get_floatingips_bound_to_host function was introduced
    recently in dvr_local_router to retrieve the external
    interface name for centralizing the floatingip.
    
    This function was throwing a 'KeyError' on fip['host'] and
    not required for centralized floatingips anymore.
    
    The get_external_device_interface_name in dvr_local_router
    will try to get the 'fg' interface that is required for
    the bound floating-ips to clear up some of the rules.
    In the case of the centralized unbound floating-ips, the
    'qg' external interface is retreived from
    get_snat_external_device_interface_name that is defined
    in 'dvr_edge_router' and based on the namespace.
    
    So _get_floatingips_bound_to_host can be removed from
    get_external_device_inteface_name.
    
    Closes-Bug: 1712412
    
    Change-Id: I94c0a071df32f572745a2c29942956c3da9f309b


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1712412

Title:
  DVR external port setup fails with KeyError: 'host'

Status in neutron:
  Fix Released

Bug description:
  http://logs.openstack.org/28/481928/9/check/gate-tempest-dsvm-neutron-
  dvr-multinode-scenario-ubuntu-xenial-
  nv/8fb7c92/logs/subnode-2/screen-q-l3.txt.gz?level=TRACE

  Aug 22 07:55:55.697082 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 02fc503f-72c9-49bd-a4f8-30305a7efb90: KeyError: 'host'
  Aug 22 07:55:55.697264 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Aug 22 07:55:55.697446 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 538, in _process_router_update
  Aug 22 07:55:55.697665 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  Aug 22 07:55:55.697850 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 475, in _process_router_if_compatible
  Aug 22 07:55:55.698048 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  Aug 22 07:55:55.698229 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 490, in _process_updated_router
  Aug 22 07:55:55.698405 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ri.process()
  Aug 22 07:55:55.698581 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 737, in process
  Aug 22 07:55:55.698756 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process()
  Aug 22 07:55:55.698936 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
  Aug 22 07:55:55.699114 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process()
  Aug 22 07:55:55.699289 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/ha_router.py", line 436, in process
  Aug 22 07:55:55.699469 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
  Aug 22 07:55:55.699654 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 189, in call
  Aug 22 07:55:55.699903 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.logger(e)
  Aug 22 07:55:55.700084 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Aug 22 07:55:55.700260 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.force_reraise()
  Aug 22 07:55:55.700457 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Aug 22 07:55:55.700635 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  Aug 22 07:55:55.700816 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/common/utils.py", line 186, in call
  Aug 22 07:55:55.700995 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  Aug 22 07:55:55.701180 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 1120, in process
  Aug 22 07:55:55.701357 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self.process_external()
  Aug 22 07:55:55.701537 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 560, in process_external
  Aug 22 07:55:55.701713 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process_external()
  Aug 22 07:55:55.701886 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 895, in process_external
  Aug 22 07:55:55.702070 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  Aug 22 07:55:55.702248 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/router_info.py", line 795, in _process_external_gateway
  Aug 22 07:55:55.702422 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     self._handle_router_snat_rules(gw_port, interface_name)
  Aug 22 07:55:55.702609 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_edge_router.py", line 184, in _handle_router_snat_rules
  Aug 22 07:55:55.702788 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ex_gw_port, interface_name)
  Aug 22 07:55:55.702980 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 514, in _handle_router_snat_rules
  Aug 22 07:55:55.703155 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     ext_device_name = self.get_external_device_interface_name(ex_gw_port)
  Aug 22 07:55:55.703329 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 444, in get_external_device_interface_name
  Aug 22 07:55:55.703503 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if not self._get_floatingips_bound_to_host(floating_ips):
  Aug 22 07:55:55.703683 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent   File "/opt/stack/new/neutron/neutron/agent/l3/dvr_local_router.py", line 550, in _get_floatingips_bound_to_host
  Aug 22 07:55:55.703856 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent     if (i['host'] == self.host or
  Aug 22 07:55:55.704042 ubuntu-xenial-2-node-rax-ord-10554784-832133 neutron-l3-agent[2539]: ERROR neutron.agent.l3.agent KeyError: 'host'

  This is probably result of
  https://review.openstack.org/#/c/437986/27/neutron/agent/l3/dvr_local_router.py
  that we sqeezed late in Pike.

  This is probably the root cause of ssh connectivity timeouts we
  observe in dvr scenario job.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1712412/+subscriptions