yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1906375] Re: [L3] router HA port concurrently deleting

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Edward Hope-Morley <1906375@xxxxxxxxxxxxxxxxxx>
Date: Fri, 11 Mar 2022 15:35:11 -0000
Reply-to: Bug 1906375 <1906375@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This fix for this is merged [1] and released to ill update the status
accordingly.

[1]
https://github.com/openstack/neutron/commit/91eb3d8346a8964aa046d1e016d571056de868de

** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1906375

Title:
  [L3] router HA port concurrently deleting

Status in neutron:
  Fix Released

Bug description:
  Router HA port may be deleted concurrently while the plugin is trying to update. Then a PortNotFound exception raised.
  ERROR was found at rocky deployment, but the master branch has the same code.

  
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server [req-86012433-ab6b-41f5-bbba-411ec3e1d973 - - - - -] Exception during message handling: PortNotFound: Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be found.
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 93, in update_all_ha_network_port_statuses
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     context, p, {'port': {'status': constants.PORT_STATUS_DOWN}})
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 632, in inner
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1420, in update_port
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     original_port = self.get_port(context, id)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     return fn(*args, **kwargs)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1462, in get_port
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     port = self._get_port(context, id)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 264, in _get_port
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server     raise n_exc.PortNotFound(port_id=id)
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server PortNotFound: Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be found.
  2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
  2020-12-01 10:53:03.921 62076 ERROR oslo_messaging.rpc.server [req-3822c366-0cb7-46c3-ad6b-b50081da3dc8 - - - - -] Exception during message handling: PortNotFound: Port 80911dcd-06be-456a-9054-4bf9f40680aa could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1906375/+subscriptions
References

[Bug 1906375] [NEW] [L3] router HA port concurrently deleting
From: LIU Yulong, 2020-12-01