← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #09954

[Bug 1281574] [NEW] nec plugin should handle OFC port-deletion race condition

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

There is a case where multiple OFC delete-port operations run in parallel. It is usually observed in tempest api tests:
ofc-delete-port triggered by delete-network API request and ofc-delete-port request from dhcp-agent (release_dhcp_port) triggered by delete-subnet.

There are several manifests I see,  however this kind of "not found" is
a valid situation and should be ignored during deleting OFC port.

http://133.242.19.163:8000/neutron-ci-logs/Neutron_Gate/FAILURES/675/
(PortNotFound)

2014-01-28 08:12:37.449 32365 ERROR neutron.api.v2.resource [req-13a26437-c201-4fc6-8af4-dfbff6663f3b None] delete failed
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in delete
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 356, in delete_network
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     port = self.deactivate_port(context, port)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 252, in deactivate_port
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     port_status)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 166, in _update_resource_status
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     obj_db = obj_getter(context, id)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 266, in _get_port
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     raise q_exc.PortNotFound(port_id=id)
2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource PortNotFound: Port b6d8480f-4c59-4095-bc72-bc7516fba1d7 could not be found

http://133.242.19.163:8000/neutron-ci-logs/Neutron_Gate/FAILURES/588/
2014-01-24 11:54:40.217 31910 DEBUG neutron.plugins.nec.common.ofc_client [-] OFC returns [202:] do_request /opt/stack/neutron/neutron/plugins/nec/common/ofc_client.py:85
2014-01-24 11:54:40.410 31910 DEBUG neutron.plugins.nec.common.ofc_client [-] OFC returns [404:] do_request /opt/stack/neutron/neutron/plugins/nec/common/ofc_client.py:85
2014-01-24 11:54:40.410 31910 WARNING neutron.plugins.nec.common.ofc_client [-] Operation on OFC failed: status=404, detail=
2014-01-24 11:54:40.411 31910 ERROR neutron.plugins.nec.nec_plugin [-] delete_ofc_port() failed due to An OFC exception has occurred: Operation on OFC failed
2014-01-24 11:54:40.523 31910 ERROR neutron.api.v2.resource [-] delete failed
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in delete
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 362, in delete_network
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     raise nexc.OFCException(reason=reason)
2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource OFCException: An OFC exception has occurred: Failed to delete port(s)=f77286eb-8493-406c-8240-6e62e027c59d from OFC.

** Affects: neutron
     Importance: Medium
     Assignee: Akihiro Motoki (amotoki)
         Status: In Progress


** Tags: nec

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1281574

Title:
  nec plugin should handle OFC port-deletion race condition

Status in OpenStack Neutron (virtual network service):
  In Progress

Bug description:
  There is a case where multiple OFC delete-port operations run in parallel. It is usually observed in tempest api tests:
  ofc-delete-port triggered by delete-network API request and ofc-delete-port request from dhcp-agent (release_dhcp_port) triggered by delete-subnet.

  There are several manifests I see,  however this kind of "not found"
  is a valid situation and should be ignored during deleting OFC port.

  http://133.242.19.163:8000/neutron-ci-logs/Neutron_Gate/FAILURES/675/
  (PortNotFound)

  2014-01-28 08:12:37.449 32365 ERROR neutron.api.v2.resource [req-13a26437-c201-4fc6-8af4-dfbff6663f3b None] delete failed
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in delete
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 356, in delete_network
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     port = self.deactivate_port(context, port)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 252, in deactivate_port
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     port_status)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 166, in _update_resource_status
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     obj_db = obj_getter(context, id)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 266, in _get_port
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource     raise q_exc.PortNotFound(port_id=id)
  2014-01-28 08:12:37.449 32365 TRACE neutron.api.v2.resource PortNotFound: Port b6d8480f-4c59-4095-bc72-bc7516fba1d7 could not be found

  http://133.242.19.163:8000/neutron-ci-logs/Neutron_Gate/FAILURES/588/
  2014-01-24 11:54:40.217 31910 DEBUG neutron.plugins.nec.common.ofc_client [-] OFC returns [202:] do_request /opt/stack/neutron/neutron/plugins/nec/common/ofc_client.py:85
  2014-01-24 11:54:40.410 31910 DEBUG neutron.plugins.nec.common.ofc_client [-] OFC returns [404:] do_request /opt/stack/neutron/neutron/plugins/nec/common/ofc_client.py:85
  2014-01-24 11:54:40.410 31910 WARNING neutron.plugins.nec.common.ofc_client [-] Operation on OFC failed: status=404, detail=
  2014-01-24 11:54:40.411 31910 ERROR neutron.plugins.nec.nec_plugin [-] delete_ofc_port() failed due to An OFC exception has occurred: Operation on OFC failed
  2014-01-24 11:54:40.523 31910 ERROR neutron.api.v2.resource [-] delete failed
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in delete
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/nec/nec_plugin.py", line 362, in delete_network
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource     raise nexc.OFCException(reason=reason)
  2014-01-24 11:54:40.523 31910 TRACE neutron.api.v2.resource OFCException: An OFC exception has occurred: Failed to delete port(s)=f77286eb-8493-406c-8240-6e62e027c59d from OFC.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1281574/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next