yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1864014] [NEW] Upgrade from Rocky to Stein, router namespace disappear

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Zhao <kevin.zhao@xxxxxxxxxx>
Date: Thu, 20 Feb 2020 09:12:09 -0000
Reply-to: Bug 1864014 <1864014@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Upgrade All-in-one from Rocky to Stein.
Upgrading finished but the router namespace disappears.

============================================================
Before:
ip netns list
qrouter-79658dd5-e3b4-4b13-a361-16d696ed1d1c (id: 1)
qdhcp-4a183162-64f5-49f9-a615-7c0fd63cf2a8 (id: 0)

After:
ip netns list
============================================================
After about 1 minutes, dhcp ns has appeared and no error on dhcp-agent,
but qrouter ns is still missing, until manually restart the docker container l3-agent.

l3-agent error after upgrade:
2020-02-20 02:57:07.306 12 INFO neutron.common.config [-] Logging enabled!
2020-02-20 02:57:07.308 12 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 14.0.4
2020-02-20 02:57:08.616 12 INFO neutron.agent.l3.agent [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Agent HA routers count 0
2020-02-20 02:57:08.619 12 INFO neutron.agent.agent_extensions_manager [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Loaded agent extensions: []
2020-02-20 02:57:08.657 12 INFO eventlet.wsgi.server [-] (12) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
2020-02-20 02:57:08.710 12 INFO neutron.agent.l3.agent [-] L3 agent started
2020-02-20 02:57:10.716 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpg8Ihqa/privsep.sock']
2020-02-20 02:57:11.750 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Spawned new privsep daemon via rootwrap
2020-02-20 02:57:11.614 29 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-02-20 02:57:11.622 29 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-02-20 02:57:11.627 29 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none
2020-02-20 02:57:11.628 29 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29
2020-02-20 02:57:14.449 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 0.000
2020-02-20 02:57:24.160 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager

2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info self.process_floating_ip_address_scope_rules()
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info self.gen.next()
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info raise l3_exc.IpTablesApplyException(msg)
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 79658dd5-e3b4-4b13-a361-16d696ed1d1c: IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 567, in _process_router_if_compatible
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self._process_added_router(router)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 575, in _process_added_router
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent ri.process()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.logger(e)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.force_reraise()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1189, in process
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.process_address_scope()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1152, in process_address_scope
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.process_floating_ip_address_scope_rules()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.gen.next()
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent raise l3_exc.IpTablesApplyException(msg)
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules
2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
2020-02-20 02:57:26.391 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 11.942
2020-02-20 02:57:27.878 12 INFO neutron.agent.linux.interface [-] Device qg-a456d6e2-1d already exists
2020-02-20 02:57:32.276 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager

** Affects: neutron
     Importance: Undecided
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1864014

Title:
  Upgrade from Rocky to Stein, router namespace disappear

Status in neutron:
  Confirmed

Bug description:
  Upgrade All-in-one from Rocky to Stein.
  Upgrading finished but the router namespace disappears.

  ============================================================
  Before:
  ip netns list
  qrouter-79658dd5-e3b4-4b13-a361-16d696ed1d1c (id: 1)
  qdhcp-4a183162-64f5-49f9-a615-7c0fd63cf2a8 (id: 0)

  After:
  ip netns list
  ============================================================
  After about 1 minutes, dhcp ns has appeared and no error on dhcp-agent,
  but qrouter ns is still missing, until manually restart the docker container l3-agent.

  l3-agent error after upgrade:
  2020-02-20 02:57:07.306 12 INFO neutron.common.config [-] Logging enabled!
  2020-02-20 02:57:07.308 12 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 14.0.4
  2020-02-20 02:57:08.616 12 INFO neutron.agent.l3.agent [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Agent HA routers count 0
  2020-02-20 02:57:08.619 12 INFO neutron.agent.agent_extensions_manager [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Loaded agent extensions: []
  2020-02-20 02:57:08.657 12 INFO eventlet.wsgi.server [-] (12) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
  2020-02-20 02:57:08.710 12 INFO neutron.agent.l3.agent [-] L3 agent started
  2020-02-20 02:57:10.716 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpg8Ihqa/privsep.sock']
  2020-02-20 02:57:11.750 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Spawned new privsep daemon via rootwrap
  2020-02-20 02:57:11.614 29 INFO oslo.privsep.daemon [-] privsep daemon starting
  2020-02-20 02:57:11.622 29 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2020-02-20 02:57:11.627 29 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none
  2020-02-20 02:57:11.628 29 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29
  2020-02-20 02:57:14.449 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 0.000
  2020-02-20 02:57:24.160 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager

  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info self.process_floating_ip_address_scope_rules()
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info self.gen.next()
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info raise l3_exc.IpTablesApplyException(msg)
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 79658dd5-e3b4-4b13-a361-16d696ed1d1c: IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 567, in _process_router_if_compatible
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self._process_added_router(router)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 575, in _process_added_router
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent ri.process()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.logger(e)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.force_reraise()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1189, in process
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.process_address_scope()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1152, in process_address_scope
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.process_floating_ip_address_scope_rules()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent self.gen.next()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent raise l3_exc.IpTablesApplyException(msg)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
  2020-02-20 02:57:26.391 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 11.942
  2020-02-20 02:57:27.878 12 INFO neutron.agent.linux.interface [-] Device qg-a456d6e2-1d already exists
  2020-02-20 02:57:32.276 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1864014/+subscriptions