yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1892866@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

L3 agent can't handle properly the case when it has to remove HA router and for some reason keepalived's config is already deleted.
It can happen e.g. in case when router is deleted, there is some error during processing this deletion so agent tries to process it again but then ha_conf/<router_id> is already removed.

There is error like:

Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent [-] Error while deleting router e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 514, in _safe_router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self._router_removed(ri, router_id)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 551, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.router_info[router_id] = ri
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.force_reraise()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     raise value
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 547, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     ri.delete()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.disable_keepalived()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in disable_keepalived
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     shutil.rmtree(conf_dir)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     onerror(os.lstat, path, sys.exc_info())
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     orig_st = os.lstat(path)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'

in such case.

** Affects: neutron
     Importance: Medium
     Assignee: Slawek Kaplonski (slaweq)
         Status: In Progress


** Tags: l3-ha

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1892866

Title:
  FileNotFoundError not cached during cleaning of HA router

Status in neutron:
  In Progress

Bug description:
  L3 agent can't handle properly the case when it has to remove HA router and for some reason keepalived's config is already deleted.
  It can happen e.g. in case when router is deleted, there is some error during processing this deletion so agent tries to process it again but then ha_conf/<router_id> is already removed.

  There is error like:

  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent [-] Error while deleting router e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 514, in _safe_router_removed
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self._router_removed(ri, router_id)
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 551, in _router_removed
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.router_info[router_id] = ri
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.force_reraise()
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     raise value
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 547, in _router_removed
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     ri.delete()
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     self.disable_keepalived()
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in disable_keepalived
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     shutil.rmtree(conf_dir)
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     onerror(os.lstat, path, sys.exc_info())
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent     orig_st = os.lstat(path)
  Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'

  in such case.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1892866/+subscriptions