yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1920968] [NEW] [OVN] Stale Logical Router Port entry fail to be removed by maintenance task

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lucas Alvares Gomes <1920968@xxxxxxxxxxxxxxxxxx>
Date: Tue, 23 Mar 2021 16:17:57 -0000
Reply-to: Bug 1920968 <1920968@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=1940724

The maintenance task invokes the delete_router_port() method from
OVNClient when it tries to delete a stale Logical_Router_Port entry but,
if the router is also gone an exception will be raised because the code
in delete_router_port() will try to fetch the router from the Neutron
database.

We need to make the delete_router_port() more resilient to those
situations, if the router is gone it means that the router port is also
gone so we just need to get rid of it in the OVN database as well.

2021-03-19 08:01:00.271 39 DEBUG futurist.periodics [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Submitting periodic callback 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronizing Neutron and OVN databases check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:341
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Number of inconsistencies found at delete: router_ports=1 _log /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:322
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Fixing resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports) at delete check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:375
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Failed to fix deleted resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports): neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 198, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     context, l3_models.Router, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 159, in get_by_id
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return query.filter(model.id == object_id).one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3282, in one
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise orm_exc.NoResultFound("No row was found for one()")
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance During handling of the above exception, another exception occurred:
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 380, in check_for_inconsistencies
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self._fix_delete(row)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 261, in _fix_delete
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     res_map['ovn_delete'](row.resource_uuid)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1502, in delete_router_port
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._l3_plugin.get_router(context, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return method(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     setattr(e, '_RETRY_EXCEEDED', True)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     ectxt.value = e.inner_exc
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*dup_args, **dup_kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 533, in get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._get_router(context, id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 200, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise l3_exc.RouterNotFound(router_id=router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.350 39 INFO networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronization finished (took 0.06 seconds)

** Affects: neutron
     Importance: High
     Assignee: Lucas Alvares Gomes (lucasagomes)
         Status: Confirmed


** Tags: ovn

** Changed in: neutron
     Assignee: (unassigned) => Lucas Alvares Gomes (lucasagomes)

** Changed in: neutron
       Status: New => Confirmed

** Changed in: neutron
   Importance: Undecided => High

** Tags added: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1920968

Title:
  [OVN] Stale Logical Router Port entry fail to be removed by
  maintenance task

Status in neutron:
  Confirmed

Bug description:
  Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=1940724

  The maintenance task invokes the delete_router_port() method from
  OVNClient when it tries to delete a stale Logical_Router_Port entry
  but, if the router is also gone an exception will be raised because
  the code in delete_router_port() will try to fetch the router from the
  Neutron database.

  We need to make the delete_router_port() more resilient to those
  situations, if the router is gone it means that the router port is
  also gone so we just need to get rid of it in the OVN database as
  well.

  2021-03-19 08:01:00.271 39 DEBUG futurist.periodics [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Submitting periodic callback 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
  2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronizing Neutron and OVN databases check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:341
  2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Number of inconsistencies found at delete: router_ports=1 _log /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:322
  2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Fixing resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports) at delete check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:375
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Failed to fix deleted resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports): neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 198, in _get_router
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     context, l3_models.Router, router_id)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 159, in get_by_id
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return query.filter(model.id == object_id).one()
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3282, in one
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise orm_exc.NoResultFound("No row was found for one()")
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance sqlalchemy.orm.exc.NoResultFound: No row was found for one()
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance During handling of the above exception, another exception occurred:
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 380, in check_for_inconsistencies
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self._fix_delete(row)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 261, in _fix_delete
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     res_map['ovn_delete'](row.resource_uuid)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1502, in delete_router_port
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._l3_plugin.get_router(context, router_id)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return method(*args, **kwargs)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     setattr(e, '_RETRY_EXCEEDED', True)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     ectxt.value = e.inner_exc
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*args, **kwargs)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     self.force_reraise()
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     six.reraise(self.type_, self.value, self.tb)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise value
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     return f(*dup_args, **dup_kwargs)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 533, in get_router
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     router = self._get_router(context, id)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance   File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 200, in _get_router
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance     raise l3_exc.RouterNotFound(router_id=router_id)
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
  2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
  2021-03-19 08:01:00.350 39 INFO networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronization finished (took 0.06 seconds)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1920968/+subscriptions
Follow ups

[Bug 1920968] Re: [OVN] Stale Logical Router Port entry fail to be removed by maintenance task
From: Brian Haley, 2022-11-08