yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1863982] Re: Upgrade from Rocky to Stein, router namespace disappear

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bernard Cafarelli <bcafarel@xxxxxxxxxx>
Date: Thu, 20 Feb 2020 17:12:10 -0000
Reply-to: Bug 1863982 <1863982@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1863982

Title:
  Upgrade from Rocky to Stein, router namespace disappear

Status in kolla-ansible:
  Confirmed
Status in neutron:
  New

Bug description:
  Upgrade All-in-one from Rocky to Stein.
  Upgrading finished but the router namespace disappears.

  ============================================================
  Before:
  ip netns list
  qrouter-79658dd5-e3b4-4b13-a361-16d696ed1d1c (id: 1)
  qdhcp-4a183162-64f5-49f9-a615-7c0fd63cf2a8 (id: 0)

  After:
  ip netns list
  ============================================================
  After about 1 minutes, dhcp ns has appeared and no error on dhcp-agent,
  but qrouter ns is still missing, until manually restart the docker container l3-agent.

  l3-agent error after upgrade:
  2020-02-20 02:57:07.306 12 INFO neutron.common.config [-] Logging enabled!
  2020-02-20 02:57:07.308 12 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 14.0.4
  2020-02-20 02:57:08.616 12 INFO neutron.agent.l3.agent [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Agent HA routers count 0
  2020-02-20 02:57:08.619 12 INFO neutron.agent.agent_extensions_manager [req-95654890-dab3-4106-b56d-c2685fb96f29 - - - - -] Loaded agent extensions: []
  2020-02-20 02:57:08.657 12 INFO eventlet.wsgi.server [-] (12) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
  2020-02-20 02:57:08.710 12 INFO neutron.agent.l3.agent [-] L3 agent started
  2020-02-20 02:57:10.716 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpg8Ihqa/privsep.sock']
  2020-02-20 02:57:11.750 12 INFO oslo.privsep.daemon [req-681aad3f-ae14-4315-b96d-5e95225cdf92 - - - - -] Spawned new privsep daemon via rootwrap
  2020-02-20 02:57:11.614 29 INFO oslo.privsep.daemon [-] privsep daemon starting
  2020-02-20 02:57:11.622 29 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2020-02-20 02:57:11.627 29 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none
  2020-02-20 02:57:11.628 29 INFO oslo.privsep.daemon [-] privsep daemon running as pid 29
  2020-02-20 02:57:14.449 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 0.000
  2020-02-20 02:57:24.160 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager


  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info     self.process_floating_ip_address_scope_rules()
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info     self.gen.next()
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info     raise l3_exc.IpTablesApplyException(msg)
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.388 12 ERROR neutron.agent.l3.router_info
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 79658dd5-e3b4-4b13-a361-16d696ed1d1c: IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 567, in _process_router_if_compatible
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 575, in _process_added_router
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     ri.process()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self.logger(e)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1189, in process
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self.process_address_scope()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1152, in process_address_scope
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self.process_floating_ip_address_scope_rules()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     self.gen.next()
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 438, in defer_apply
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent     raise l3_exc.IpTablesApplyException(msg)
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules
  2020-02-20 02:57:26.389 12 ERROR neutron.agent.l3.agent
  2020-02-20 02:57:26.391 12 INFO neutron.agent.l3.agent [-] Starting router update for 79658dd5-e3b4-4b13-a361-16d696ed1d1c, action 3, priority 2, update_id 49908db7-8a8c-410f-84a7-9e95a3dede16. Wait time elapsed: 11.942
  2020-02-20 02:57:27.878 12 INFO neutron.agent.linux.interface [-] Device qg-a456d6e2-1d already exists
  2020-02-20 02:57:32.276 12 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1863982/+subscriptions