← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #12215

[Bug 1297992] [NEW] deleting an interface hits a race condition and fails

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

We are seeing nova REST calls to delete  an attached interface fail to work:
  DELETE http://127.0.0.1:8774/v3/servers/UUID/os-attach-interfaces/UUID
  DELETE http://127.0.0.1:8774/v2/UUID/servers/UUID/os-interface/UUID


In nova-cpu we see 

DEBUG neutronclient.client [-] RESP:{'status': '200', 'content-length': '13', 'content-location': 'http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp', 'date': 'Wed, 26 Mar 2014 00:44:04 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-59333b8f-072d-4bce-b5cc-03d9af48dcc8'} {"ports": []}
/python-neutronclient/neutronclient/common/utils.py:179
 nova.network.api [req-61feb3de-431b-4dcc-81c7-8a8e5b6b03ed AttachInterfacesV3Test-673399961 AttachInterfacesV3Test-295185211] Updating cache with info: [VIF({'ovs_interfaceid': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.4'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap530319a7-b4', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:e8:72:f4', 'active': True, 'type': u'ovs', 'id': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'qbg_params': None}), VIF({'ovs_interfaceid': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.2'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap8077e5f2-e6', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:83:34:a9', 'active': True, 'type': u'ovs', 'id': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'qbg_params': None})] update_instance_cache_with_nw_info /opt/stack/new/nova/nova/network/api.py:74
ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
TRACE oslo.messaging.rpc.dispatcher     incoming.message))
TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function
TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface
TRACE oslo.messaging.rpc.dispatcher     "attached") % port_id)
TRACE oslo.messaging.rpc.dispatcher PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
TRACE oslo.messaging.rpc.dispatcher 
ERROR oslo.messaging._drivers.common [-] Returning exception Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached to caller
ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface\n    "attached") % port_id)\n', 'PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached\n']

http://logs.openstack.org/22/82922/2/gate/gate-tempest-dsvm-
neutron/87c67c5/logs/screen-n-cpu.txt.gz#_2014-03-26_00_44_04_250


It looks like when nova does 'curl -i http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp' neutron is returning an empty list.


logstash query: message:"PortNotFound: Port" AND message:"is not attached" AND filename:logs*screen-n-cpu.txt

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1297992

Title:
  deleting an interface hits a race condition and fails

Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  New

Bug description:
  We are seeing nova REST calls to delete  an attached interface fail to work:
    DELETE http://127.0.0.1:8774/v3/servers/UUID/os-attach-interfaces/UUID
    DELETE http://127.0.0.1:8774/v2/UUID/servers/UUID/os-interface/UUID

  
  In nova-cpu we see 

  DEBUG neutronclient.client [-] RESP:{'status': '200', 'content-length': '13', 'content-location': 'http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp', 'date': 'Wed, 26 Mar 2014 00:44:04 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-59333b8f-072d-4bce-b5cc-03d9af48dcc8'} {"ports": []}
  /python-neutronclient/neutronclient/common/utils.py:179
   nova.network.api [req-61feb3de-431b-4dcc-81c7-8a8e5b6b03ed AttachInterfacesV3Test-673399961 AttachInterfacesV3Test-295185211] Updating cache with info: [VIF({'ovs_interfaceid': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.4'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap530319a7-b4', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:e8:72:f4', 'active': True, 'type': u'ovs', 'id': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'qbg_params': None}), VIF({'ovs_interfaceid': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.2'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap8077e5f2-e6', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:83:34:a9', 'active': True, 'type': u'ovs', 'id': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'qbg_params': None})] update_instance_cache_with_nw_info /opt/stack/new/nova/nova/network/api.py:74
  ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
  TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function
  TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface
  TRACE oslo.messaging.rpc.dispatcher     "attached") % port_id)
  TRACE oslo.messaging.rpc.dispatcher PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
  TRACE oslo.messaging.rpc.dispatcher 
  ERROR oslo.messaging._drivers.common [-] Returning exception Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached to caller
  ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface\n    "attached") % port_id)\n', 'PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached\n']

  http://logs.openstack.org/22/82922/2/gate/gate-tempest-dsvm-
  neutron/87c67c5/logs/screen-n-cpu.txt.gz#_2014-03-26_00_44_04_250

  
  It looks like when nova does 'curl -i http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp' neutron is returning an empty list.

  
  logstash query: message:"PortNotFound: Port" AND message:"is not attached" AND filename:logs*screen-n-cpu.txt

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1297992/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next