← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1700583] Re: No volume Block Device Mapping in assisted snapshot with Quobyte

 

Verified builds are working again, even on old and previously failing
changes. This means the issue was somewhere in the DevStack/tempest/...
env and was solved yesterday by somebody. :)

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1700583

Title:
  No volume Block Device Mapping in assisted snapshot with Quobyte

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Since roughly last friday Quobyte CIs with a Cinder / Nova setup fail with an error during assisted snapshots in four tests:
  test_create_ebs_image_and_check_boot|test_snapshot_create_delete_with_volume_in_use|test_snapshot_create_offline_delete_online|test_volume_boot_pattern

  The error declares that the volume to be snapshotted has no block
  device mapping.

  I currently cannot identify if this is a Nova or Cinder issue, so
  please keep an open mind when looking into this.

  Complete example CI test runs with all logs can be found at e.g.:

  
  http://78.46.57.153:8081/refs-changes-26-475226-3/
  http://78.46.57.153:8081/refs-changes-02-476402-2/

  
  n-api log excerpt:
  [...]
  2017-06-23 14:24:18.137 10730 DEBUG nova.api.openstack.wsgi [req-484e4bd2-e091-4411-9252-38549f7c9dbd admin admin] Action: 'create', calling method: <bound method AssistedVolumeSnapshotsController.create of <nova.api.openstack.compute.assisted_volume_snapshots.AssistedVolumeSnapshotsController object at 0x7fdd470e8fd0>>, body: {"snapshot": {"create_info": {"snapshot_id": "30bd2ad6-37a0-4027-a24e-00657adb0cca", "type": "qcow2", "new_file": "volume-43245219-bd54-4b03-a371-cca5836e58e5.30bd2ad6-37a0-4027-a24e-00657adb0cca"}, "volume_id": "43245219-bd54-4b03-a371-cca5836e58e5"}} _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:609
  2017-06-23 14:24:18.438 10730 INFO nova.api.openstack.wsgi [req-484e4bd2-e091-4411-9252-38549f7c9dbd admin admin] HTTP exception thrown: No volume Block Device Mapping with id 43245219-bd54-4b03-a371-cca5836e58e5.: HTTPBadRequest: No volume Block Device Mapping with id 43245219-bd54-4b03-a371-cca5836e58e5.
  2017-06-23 14:24:18.440 10730 DEBUG nova.api.openstack.wsgi [req-484e4bd2-e091-4411-9252-38549f7c9dbd admin admin] Returning 400 to user: No volume Block Device Mapping with id 43245219-bd54-4b03-a371-cca5836e58e5. __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029
  2017-06-23 14:24:18.441 10730 INFO nova.osapi_compute.wsgi.server [req-484e4bd2-e091-4411-9252-38549f7c9dbd admin admin] 127.0.0.1 "POST /v2.1/os-assisted-volume-snapshots HTTP/1.1" status: 400 len: 543 time: 0.6530399
  2017-06-23 14:24:18.770 10730 INFO nova.osapi_compute.wsgi.server [req-a0c161b6-12ba-4dbf-aa10-5fa34565441c tempest-TestVolumeBootPattern-1485741991 tempest-TestVolumeBootPattern-1485741991] 127.0.0.1 "GET /v2.1/servers/0fcae274-2d09-45e7-ba84-df2b53526895 HTTP/1.1" status: 200 len: 1762 time: 0.8687029
  [...]

  
  Reading the test logs I see for instance that this error is thrown right after the volume attachment 
  to the VM is successful. I'd expect the snapshot API call to be able to find the mapping in that case. Example snippet:

  
   [... attaching volume to VM]

      2017-06-26 13:13:43,182 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097 0.223s
      2017-06-26 13:13:43,182 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
              Body: None
          Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097', u'x-compute-request-id': 'req-fc80f9a0-dd4c-4e9d-b044-39007bbcdbae', u'connection': 'close', 'status': '200', u'content-length': '904', u'x-openstack-request-id': 'req-fc80f9a0-dd4c-4e9d-b044-39007bbcdbae', u'date': 'Mon, 26 Jun 2017
   13:13:43 GMT'}
              Body: {"volume": {"status": "attaching", "user_id": "8fa70618b61e4290908a964b6c8d11d9", "attachments": [], "links": [{"href": "http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "self"}, {"href": "http://127.0.0.1:8776/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "bookmark"}], "availability_zone": "nova", "bootable": "false", "encrypted": false, "created_at": "2017-06-26T13:13:29.000000", "description": null, "os-vol-tenant-attr:tenant_id": "182a9178e3334660ad978457a2aecc9d", "updated_at": "2017-06-26T13:13:41.000000", "volume_type": "Quobyte", "name": "tempest-VolumesSnapshotTestJSON-Volume-1014042071", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {}, "id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "size": 1}}
      2017-06-26 13:13:44,333 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097 0.148s
      2017-06-26 13:13:44,346 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None                                                                                                                                                                                                                                                                                                                                                                                                                           Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097', u'x-compute-request-id': 'req-abc6354c-5edb-488e-9c07-e53012afe6ea', u'connection': 'close', 'status': '200', u'content-length': '904', u'x-openstack-request-id': 'req-abc6354c-5edb-488e-9c07-e53012afe6ea', u'date': 'Mon, 26 Jun 2017 13:13:44 GMT'}                                                                                                                                                                                                                                                                                                                                                                                                                                      Body: {"volume": {"status": "attaching", "user_id": "8fa70618b61e4290908a964b6c8d11d9", "attachments": [], "links": [{"href": "http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "self"}, {"href": "http://127.0.0.1:8776/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "bookmark"}], "availability_zone": "nova", "bootable": "false", "encrypted": false, "created_at": "2017-06-26T13:13:29.000000", "description": null, "os-vol-tenant-attr:tenant_id": "182a9178e3334660ad978457a2aecc9d", "updated_at": "2017-06-26T13:13:41.000000", "volume_type": "Quobyte", "name": "tempest-VolumesSnapshotTestJSON-Volume-1014042071", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {}, "id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "size": 1}}
      2017-06-26 13:13:45,462 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097 0.114s

  
  [Attach done, volume ist listed as existing attachement to VM.]

  
      2017-06-26 13:13:45,462 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None                                                                                                                                                                                                                                                                                                                                                                                                                           Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097', u'x-compute-request-id': 'req-188ae214-1572-4c50-96da-4fba9797a924', u'connection': 'close', 'status': '200', u'content-length': '1238', u'x-openstack-request-id': 'req-188ae214-1572-4c50-96da-4fba9797a924', u'date': 'Mon, 26 Jun 2017 13:13:45 GMT'}                                                                                                                                                                                                                                                                                                                                                                                                                                     Body: {"volume": {"status": "in-use", "user_id": "8fa70618b61e4290908a964b6c8d11d9", "attachments": [{"server_id": "97e84961-87c8-4672-8e72-3f8e652d5d0e", "attachment_id": "1b172300-aa07-4764-a75f-aac567695133", "attached_at": "2017-06-26T13:13:45.000000", "host_name": null, "volume_id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "device": "/dev/vdb", "id": "df77bf3f-8602-4f28-a999-f4c73a3e9097"}], "links": [{"href": "http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "self"}, {"href": "http://127.0.0.1:8776/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097";, "rel": "bookmark"}], "availability_zone": "nova", "bootable": "false", "encrypted": false, "created_at": "2017-06-26T13:13:29.000000", "description": null, "os-vol-tenant-attr:tenant_id": "182a9178e3334660ad978457a2aecc9d", "updated_at": "2017-06-26T13:13:45.000000", "volume_type": "Quobyte", "name": "tempest-VolumesSnapshotTestJSON-Volume-1014042071", "replication_status": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "multiattach": false, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "size": 1}}
      

  [Trigger snapshot #2 from tempest test (while volume is attached to
  VM). At first the new snapshot is in 'creating' status.]

  
      2017-06-26 13:13:46,065 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 202 POST http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots 0.603s
      2017-06-26 13:13:46,066 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
              Body: {"snapshot": {"force": true, "name": "tempest-VolumesSnapshotTestJSON-Snapshot-661967408", "volume_id": "df77bf3f-8602-4f28-a999-f4c73a3e9097"}}
          Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots', u'x-compute-request-id': 'req-876cadf4-170f-42c3-87f1-a2956e6452bf', u'connection': 'close', 'status': '202', u'content-length': '309', u'x-openstack-request-id': 'req-876cadf4-170f-42c3-87f1-a2956e6452bf', u'date': 'Mon, 26 Jun 2017 13:13:46 GMT'}
              Body: {"snapshot": {"status": "creating", "size": 1, "metadata": {}, "name": "tempest-VolumesSnapshotTestJSON-Snapshot-661967408", "volume_id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "created_at": "2017-06-26T13:13:45.770696", "description": null, "id": "e6b01eec-4618-49ee-af96-d3776e881bb9", "updated_at": null}}
      2017-06-26 13:13:46,146 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots/e6b01eec-4618-49ee-af96-d3776e881bb9 0.079s
      2017-06-26 13:13:46,146 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None                                                                                                                                                                                                                                                                                                                                                                                                                           Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots/e6b01eec-4618-49ee-af96-d3776e881bb9', u'x-compute-request-id': 'req-9aee4d62-5344-4818-887a-f2b661b792e6', u'connection': 'close', 'status': '200', u'content-length': '441', u'x-openstack-request-id': 'req-9aee4d62-5344-4818-887a-f2b661b792e6', u'date': 'Mon, 26 Jun 2017 13:13:46 GMT'}                                                                                                                                                                                                                                                                                                                                                                                                                                    Body: {"snapshot": {"status": "creating", "description": null, "updated_at": null, "volume_id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "id": "e6b01eec-4618-49ee-af96-d3776e881bb9", "size": 1, "os-extended-snapshot-attributes:progress": "0%", "name": "tempest-VolumesSnapshotTestJSON-Snapshot-661967408", "os-extended-snapshot-attributes:project_id": "182a9178e3334660ad978457a2aecc9d", "created_at": "2017-06-26T13:13:46.000000", "metadata": {}}}
      2017-06-26 13:13:47,199 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:test_snapshot_create_offline_delete_online): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots/e6b01eec-4618-49ee-af96-d3776e881bb9 0.052s

  
  [Snapshot status changes from 'creating' to 'error']

  
      2017-06-26 13:13:47,199 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None                                                                                                                                                                                                                                                                                                                                                                                                                           Response - Headers: {u'content-type': 'application/json', 'content-location': 'http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/snapshots/e6b01eec-4618-49ee-af96-d3776e881bb9', u'x-compute-request-id': 'req-6ad9b5c9-019a-4f73-83b9-dbbd6befafa5', u'connection': 'close', 'status': '200', u'content-length': '462', u'x-openstack-request-id': 'req-6ad9b5c9-019a-4f73-83b9-dbbd6befafa5', u'date': 'Mon, 26 Jun 2017 13:13:47 GMT'}                                                                                                                                                                                                                                                                                                                                                                                                                                    Body: {"snapshot": {"status": "error", "description": null, "updated_at": "2017-06-26T13:13:47.000000", "volume_id": "df77bf3f-8602-4f28-a999-f4c73a3e9097", "id": "e6b01eec-4618-49ee-af96-d3776e881bb9", "size": 1, "os-extended-snapshot-attributes:progress": "0%", "name": "tempest-VolumesSnapshotTestJSON-Snapshot-661967408", "os-extended-snapshot-attributes:project_id": "182a9178e3334660ad978457a2aecc9d", "created_at": "2017-06-26T13:13:46.000000", "metadata": {}}}
      2017-06-26 13:13:47,569 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:_run_cleanups): 202 DELETE http://127.0.0.1:8774/v2.1/servers/97e84961-87c8-4672-8e72-3f8e652d5d0e/os-volume_attachments/df77bf3f-8602-4f28-a999-f4c73a3e9097 0.368s
      2017-06-26 13:13:47,570 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None                                                                                                                                                                                                                                                                                                                                                                                                                           Response - Headers: {u'content-type': 'application/json', u'x-openstack-nova-api-version': '2.1', u'vary': 'X-OpenStack-Nova-API-Version', 'content-location': 'http://127.0.0.1:8774/v2.1/servers/97e84961-87c8-4672-8e72-3f8e652d5d0e/os-volume_attachments/df77bf3f-8602-4f28-a999-f4c73a3e9097', u'openstack-api-version': 'compute 2.1', u'content-length': '0', u'connection': 'close', 'status': '202', u'x-compute-request-id': 'req-bad4db4f-cc78-40ff-a792-e99fb915e54a', u'x-openstack-request-id': 'req-bad4db4f-cc78-40ff-a792-e99fb915e54a', u'date': 'Mon, 26 Jun 2017 13:13:47 GMT'}
              Body: 
      2017-06-26 13:13:47,821 28633 INFO     [tempest.lib.common.rest_client] Request (VolumesSnapshotTestJSON:_run_cleanups): 200 GET http://127.0.0.1:8776/v2/182a9178e3334660ad978457a2aecc9d/volumes/df77bf3f-8602-4f28-a999-f4c73a3e9097 0.248s
      2017-06-26 13:13:47,822 28633 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}                                                                                                                                                                                                                                                       Body: None

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1700583/+subscriptions


References