yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1401110] Re: Volume Block Device Mapping cannot be found

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eric Harney <1401110@xxxxxxxxxxxxxxxxxx>
Date: Wed, 10 Dec 2014 21:57:50 -0000
Reply-to: Bug 1401110 <1401110@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: tempest
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1401110

Title:
   Volume Block Device Mapping cannot be found

Status in OpenStack Compute (Nova):
  Confirmed
Status in Tempest:
  New

Bug description:
  In Short:
  Block Device Mapping cannot be found for Tempest online snapshot test on devstack with remote file system driver.

  More Detailed:
  Testing Environment is plain kilo head devstack with the included Tempest testsuite and a new remotefs based cinder driver.
  The test:
       tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use
  produces an error in Nova, stacktrace:

  2014-12-10 11:14:42.329 ^[[00;32mDEBUG nova.api.openstack.wsgi [^[[01;36mreq-5a0e0e46-2455-41ee-a36a-6c14de0d1627 ^[[00;36mVolumesV1SnapshotTestJSON-2078362258 VolumesV1SnapshotTestJSON-1658640019^[[00;32m] ^[[01;35m^[[00;32mCalling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7fbd794a0690>>'^[[00m ^[[00;33mfrom (pid=12485) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:963^[[00m^M
  2014-12-10 11:14:42.374 ^[[00;36mINFO nova.osapi_compute.wsgi.server [^[[01;36mreq-5a0e0e46-2455-41ee-a36a-6c14de0d1627 ^[[00;36mVolumesV1SnapshotTestJSON-2078362258 VolumesV1SnapshotTestJSON-1658640019^[[00;36m] ^[[01;35m^[[00;36m192.168.122.67 "GET /v2/e152b27ee9d641e6a57bec75ab29c924/servers/4fb9afd5-18a9-416e-b955-d161ebbc436f HTTP/1.1" status: 200 len: 1609 time: 0.0979590^[[00m^M
  2014-12-10 11:14:43.381 ^[[00;32mDEBUG nova.api.openstack.wsgi [^[[01;36mreq-02efe9e9-c7cb-46bf-8ab4-0b70a9b84f6d ^[[00;36mVolumesV1SnapshotTestJSON-2078362258 VolumesV1SnapshotTestJSON-1658640019^[[00;32m] ^[[01;35m^[[00;32mCalling method '<bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x7fbd794a0690>>'^[[00m ^[[00;33mfrom (pid=12487) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:963^[[00m^M
  2014-12-10 11:14:43.423 ^[[00;36mINFO nova.osapi_compute.wsgi.server [^[[01;36mreq-02efe9e9-c7cb-46bf-8ab4-0b70a9b84f6d ^[[00;36mVolumesV1SnapshotTestJSON-2078362258 VolumesV1SnapshotTestJSON-1658640019^[[00;36m] ^[[01;35m^[[00;36m192.168.122.67 "GET /v2/e152b27ee9d641e6a57bec75ab29c924/servers/4fb9afd5-18a9-416e-b955-d161ebbc436f HTTP/1.1" status: 200 len: 1749 time: 0.0428371^[[00m^M
  2014-12-10 11:14:43.995 ^[[00;32mDEBUG nova.api.openstack.wsgi [^[[01;36mreq-023897db-cfd9-4259-b7c4-3621722a5eec ^[[00;36mVolumesV2SnapshotTestJSON-1786381329 VolumesV2SnapshotTestJSON-2048796393^[[00;32m] ^[[01;35m^[[00;32mAction: 'create', calling method: <bound method AssistedVolumeSnapshotsController.create of <nova.api.openstack.compute.contrib.assisted_volume_snapshots.AssistedVolumeSnapshotsController object at 0x7fbd792ef490>>, body: {"snapshot": {"create_info": {"snapshot_id": "cf916b1a-e797-41e3-bde8-39c9615aa622", "type": "qcow2", "new_file": "volume-9f7ec09d-4ff2-4ef9-96dc-2bdc07471a65.cf916b1a-e797-41e3-bde8-39c9615aa622"}, "volume_id": "9f7ec09d-4ff2-4ef9-96dc-2bdc07471a65"}}^[[00m ^[[00;33mfrom (pid=12487) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:960^[[00m^M
  2014-12-10 11:14:43.995 ^[[01;36mAUDIT nova.api.openstack.compute.contrib.assisted_volume_snapshots [^[[01;36mreq-023897db-cfd9-4259-b7c4-3621722a5eec ^[[00;36mVolumesV2SnapshotTestJSON-1786381329 VolumesV2SnapshotTestJSON-2048796393^[[01;36m] ^[[01;35m^[[01;36mCreate assisted snapshot from volume 9f7ec09d-4ff2-4ef9-96dc-2bdc07471a65^[[00m^M
  2014-12-10 11:14:44.007 ^[[01;31mERROR nova.api.openstack [^[[01;36mreq-023897db-cfd9-4259-b7c4-3621722a5eec ^[[00;36mVolumesV2SnapshotTestJSON-1786381329 VolumesV2SnapshotTestJSON-2048796393^[[01;31m] ^[[01;35m^[[01;31mCaught error: No volume Block Device Mapping with id 9f7ec09d-4ff2-4ef9-96dc-2bdc07471a65.^[[00m^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00mTraceback (most recent call last):^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return req.get_response(self.application)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    application, catch_exc_info=False)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    app_iter = application(self.environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return resp(environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 793, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return self._call_app(env, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 726, in _call_app^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return self._app(env, _fake_start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return resp(environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return resp(environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    response = self.app(environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return resp(environ, start_response)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    resp = self.call_func(req, *args, **self.kwargs)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return self.func(req, *args, **kwargs)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 936, in __call__^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    content_type, body, accept)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1001, in _process_stack^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    action_result = self.dispatch(meth, request, action_args)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1086, in dispatch^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return method(req=request, **action_args)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/api/openstack/compute/contrib/assisted_volume_snapshots.py", line 71, in create^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    create_info)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/compute/api.py", line 226, in wrapped^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    return func(self, context, target, *args, **kwargs)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/compute/api.py", line 3263, in volume_snapshot_create^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    context, volume_id, expected_attrs=['instance'])^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/objects/base.py", line 156, in wrapper^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    result = fn(cls, context, *args, **kwargs)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m  File "/opt/stack/nova/nova/objects/block_device.py", line 164, in get_by_volume_id^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m    raise exception.VolumeBDMNotFound(volume_id=volume_id)^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00mVolumeBDMNotFound: No volume Block Device Mapping with id 9f7ec09d-4ff2-4ef9-96dc-2bdc07471a65.^M
  ^[[01;31m2014-12-10 11:14:44.007 TRACE nova.api.openstack ^[[01;35m^[[00m^M

  Testing the tests individual steps via manual cli commands (create instance, attach volume, create online snapshot) works without errors in any of the involved components.
  The error occurs with our Quobyte Driver (in development, see https://blueprints.launchpad.net/cinder/+spec/quobyte-usp-driver) and apparently with the existing GlusterFS driver. Both are closely related (Quobyte build on top of GlusterFS impl) and derived from the remotefs driver. I don't know if other setups are affected.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1401110/+subscriptions
References

[Bug 1401110] [NEW] Volume Block Device Mapping cannot be found
From: Silvan Kaiser, 2014-12-10