yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1657981] Re: FloatingIPs not reachable after restart of compute node (DVR)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Swaminathan Vasudevan <souminathan@xxxxxxxxx>
Date: Mon, 12 Mar 2018 22:21:18 -0000
Reply-to: Bug 1657981 <1657981@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1657981

Title:
  FloatingIPs not reachable after restart of compute node (DVR)

Status in neutron:
  Invalid

Bug description:
  I am running OpenStack Newton on Ubuntu 16.04 using DVR. When I
  restart a compute node, the FloatingIPs of those vms running on this
  node are unreachable. A manual restart of the service
  "neutron-l3-agent" or "neutron-vpn-agent" running in on node solves
  the issue.

  I think there must be a race condition at startup.

  I get the following error in the neutron-vpn-agent.log:
  2017-01-20 07:04:52.379 2541 INFO neutron.common.config [-] Logging enabled!
  2017-01-20 07:04:52.379 2541 INFO neutron.common.config [-] /usr/bin/neutron-vpn-agent version 9.0.0
  2017-01-20 07:04:52.380 2541 WARNING stevedore.named [-] Could not load neutron.agent.linux.interface.OVSInterfaceDriver
  2017-01-20 07:04:53.112 2541 WARNING stevedore.named [req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Could not load neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
  2017-01-20 07:04:53.127 2541 INFO neutron.agent.agent_extensions_manager [req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Loaded agent extensions: ['fwaas']
  2017-01-20 07:04:53.128 2541 INFO neutron.agent.agent_extensions_manager [req-a9e10331-51ab-4c67-bfdd-0f6296510594 - - - - -] Initializing agent extension 'fwaas'
  2017-01-20 07:04:53.163 2541 WARNING oslo_config.cfg [req-bdd95fb9-bcd7-473e-a350-3bd8d6be8758 - - - - -] Option "external_network_bridge" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
  2017-01-20 07:04:53.165 2541 WARNING stevedore.named [req-bdd95fb9-bcd7-473e-a350-3bd8d6be8758 - - - - -] Could not load neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver
  2017-01-20 07:04:53.236 2541 INFO eventlet.wsgi.server [-] (2541) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
  2017-01-20 07:04:53.261 2541 INFO neutron.agent.l3.agent [-] Agent has just been revived. Doing a full sync.
  2017-01-20 07:04:53.373 2541 INFO neutron.agent.l3.agent [-] L3 agent started
  2017-01-20 07:05:22.832 2541 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"

  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 239, in call
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1062, in process
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     self.process_external(agent)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 515, in process_external
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     self.create_dvr_fip_interfaces(ex_gw_port)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 546, in create_dvr_fip_interfaces
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     self.fip_ns.update_gateway_port(fip_agent_port)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_fip_ns.py", line 239, in update_gateway_port
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     ipd.route.add_gateway(gw_ip)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 702, in add_gateway
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     self._as_root([ip_version], tuple(args))
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 373, in _as_root
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in execute
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info 
  2017-01-20 07:05:22.833 2541 ERROR neutron.agent.l3.router_info 
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 552f5cfc-27c4-4a59-842a-b3e471a94ca1
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 508, in _process_router_update
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 443, in _process_router_if_compatible
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 451, in _process_added_router
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     ri.process(self)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 580, in process
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process(agent)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process(agent)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 242, in call
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self.logger(e)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self.force_reraise()
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 239, in call
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1062, in process
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self.process_external(agent)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 515, in process_external
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self.create_dvr_fip_interfaces(ex_gw_port)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 546, in create_dvr_fip_interfaces
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self.fip_ns.update_gateway_port(fip_agent_port)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_fip_ns.py", line 239, in update_gateway_port
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     ipd.route.add_gateway(gw_ip)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 702, in add_gateway
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     self._as_root([ip_version], tuple(args))
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 373, in _as_root
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     use_root_namespace=use_root_namespace)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in execute
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "fg-67afaa06-bb"
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent 
  2017-01-20 07:05:22.929 2541 ERROR neutron.agent.l3.agent 
  2017-01-20 07:05:22.978 2541 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: arping: Device fg-67afaa06-bb not available.

  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 10.30.216.131 on fg-67afaa06-bb in namespace fip-5d873120-a1f8-4b9d-83fb-96f20fdfa9bd
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib Traceback (most recent call last):
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 1007, in _arping
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib     ip_wrapper.netns.execute(arping_cmd, check_exit_code=True)
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 894, in execute
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib     log_fail_as_error=log_fail_as_error, **kwargs)
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 138, in execute
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib     raise RuntimeError(msg)
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: arping: Device fg-67afaa06-bb not available.
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib 
  2017-01-20 07:05:22.980 2541 ERROR neutron.agent.linux.ip_lib

  After restart:
  2017-01-20 07:15:10.579 5069 INFO neutron.common.config [-] Logging enabled!
  2017-01-20 07:15:10.580 5069 INFO neutron.common.config [-] /usr/bin/neutron-vpn-agent version 9.0.0
  2017-01-20 07:15:10.581 5069 WARNING stevedore.named [-] Could not load neutron.agent.linux.interface.OVSInterfaceDriver
  2017-01-20 07:15:10.810 5069 WARNING stevedore.named [req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Could not load neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
  2017-01-20 07:15:10.812 5069 INFO neutron.agent.agent_extensions_manager [req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Loaded agent extensions: ['fwaas']
  2017-01-20 07:15:10.813 5069 INFO neutron.agent.agent_extensions_manager [req-99af950d-c09f-4f63-bfb1-5368d01f7a76 - - - - -] Initializing agent extension 'fwaas'
  2017-01-20 07:15:10.820 5069 WARNING oslo_config.cfg [req-016c1195-a7a2-4b94-b260-66c655ad4b2d - - - - -] Option "external_network_bridge" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
  2017-01-20 07:15:10.821 5069 WARNING stevedore.named [req-016c1195-a7a2-4b94-b260-66c655ad4b2d - - - - -] Could not load neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver
  2017-01-20 07:15:10.857 5069 INFO eventlet.wsgi.server [-] (5069) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
  2017-01-20 07:15:10.950 5069 INFO neutron.agent.l3.agent [-] L3 agent started

  Since now, the floating ips become reachable after a short period of
  time.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1657981/+subscriptions
References

[Bug 1657981] [NEW] FloatingIPs not reachable after restart of compute node (DVR)
From: Jens Offenbach, 2017-01-20