yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1627183] [NEW] Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", u'code': 500

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alex Ortiz <amortiz@xxxxxxxxxx>
Date: Fri, 23 Sep 2016 20:04:37 -0000
Reply-to: Bug 1627183 <1627183@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Ran tempest, hit this error. I can't get to the system right now to get
additional info. If i can, i will add more info.

19:42:37
19:42:37 ==============================
19:42:37 Failed 1 tests - output below:
19:42:37 ==============================
19:42:37
19:42:37 tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
19:42:37 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
19:42:37
19:42:37 Captured traceback:
19:42:37 ~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py", line 80, in _unrescue
19:42:37         server_id, 'ACTIVE')
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/common/waiters.py", line 75, in wait_for_server_status
19:42:37         server_id=server_id)
19:42:37     tempest.exceptions.BuildErrorException: Server 04e4aa2a-13ed-44ca-a275-bbb703c12493 failed to build and is in ERROR status
19:42:37     Details: {u'message': u"Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", u'code': 500, u'created': u'2016-09-23T19:39:46Z'}
19:42:37
19:42:37
19:42:37 Captured pythonlogging:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~~~
19:42:37     2016-09-23 19:39:36,692 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes 0.411s
19:42:37     2016-09-23 19:39:36,692 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"volume": {"display_name": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "size": 1}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes', 'x-compute-request-id': 'req-beb5ecea-c1dc-467e-9381-6c43be677b2f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.489134", "size": 1}}
19:42:37     2016-09-23 19:39:36,817 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.122s
19:42:37     2016-09-23 19:39:36,817 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-fa69239e-7fe6-4b8e-b3e9-786eb994a9f6', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:37,937 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.116s
19:42:37     2016-09-23 19:39:37,938 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-b1ec1378-2e49-48cc-aaac-6f5d7218624f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:37 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "available", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:38,329 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments 0.388s
19:42:37     2016-09-23 19:39:38,330 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '194', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments', 'x-compute-request-id': 'req-7f079443-51fb-4f41-8797-0e542d6e2995', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
19:42:37     2016-09-23 19:39:38,465 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.134s
19:42:37     2016-09-23 19:39:38,466 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-586a4fca-99cf-4c35-9c7c-80b23ecf7faf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "attaching", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:39,595 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.125s
19:42:37     2016-09-23 19:39:39,596 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '570', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-17bbc784-e95d-437c-9d20-f0a528a0762f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "in-use", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:39,833 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action 0.234s
19:42:37     2016-09-23 19:39:39,833 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"rescue": {"adminPass": "X2!oCgz87DGWc8g"}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '32', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action', 'x-compute-request-id': 'req-e93333a1-c339-4137-8363-1ec692e93594', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"adminPass": "X2!oCgz87DGWc8g"}
19:42:37     2016-09-23 19:39:40,069 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.234s
19:42:37     2016-09-23 19:39:40,069 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-98335fbb-f98d-44d2-b02a-314a6687e20e', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:40 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:41,306 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.228s
19:42:37     2016-09-23 19:39:41,306 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-fe829b17-4da3-44de-bbb9-6839c6fb4beb', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:41 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:42,541 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.227s
19:42:37     2016-09-23 19:39:42,542 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-06e65562-36ac-440b-9b38-45eb5322fa43', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:42 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:43,799 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.243s
19:42:37     2016-09-23 19:39:43,799 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-c693fccb-0153-4737-8192-d9869122659a', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:43 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:45,058 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.251s
19:42:37     2016-09-23 19:39:45,058 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1767', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-be1c970e-b954-4d73-9b75-23820d0aff25', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "RESCUE", "updated": "2016-09-23T19:39:44Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:45,064 18008 INFO     [tempest.common.waiters] State transition "ACTIVE/rescuing" ==> "RESCUE/None" after 5 second wait
19:42:37     2016-09-23 19:39:45,254 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 409 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780 0.188s
19:42:37     2016-09-23 19:39:45,254 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '409', 'content-length': '152', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-e572880b-7dcd-457d-a06a-ed004351adaf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"conflictingRequest": {"message": "Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state rescued", "code": 409}}
19:42:37     2016-09-23 19:39:45,396 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 202 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action 0.140s
19:42:37     2016-09-23 19:39:45,396 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"unrescue": {}}
19:42:37         Response - Headers: {'status': '202', 'content-length': '0', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action', 'x-compute-request-id': 'req-7010ff1d-7d7c-4052-a2d5-3cddb6ddee4c', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body:
19:42:37     2016-09-23 19:39:45,671 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.274s
19:42:37     2016-09-23 19:39:45,672 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '1775', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-8069154d-767b-4d75-ab32-83c8d07ae745', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "unrescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "RESCUE", "updated": "2016-09-23T19:39:45Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:46,921 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.240s
19:42:37     2016-09-23 19:39:46,921 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '2010', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-99919910-98c2-40bb-90fd-0828ead5f1bf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:46 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ERROR", "updated": "2016-09-23T19:39:46Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "fault": {"message": "Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", "code": 500, "created": "2016-09-23T19:39:46Z"}, "config_drive": ""}}
19:42:37     2016-09-23 19:39:46,928 18008 INFO     [tempest.common.waiters] State transition "RESCUE/unrescuing" ==> "ERROR/None" after 1 second wait
19:42:37     2016-09-23 19:39:47,151 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 409 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780 0.222s
19:42:37     2016-09-23 19:39:47,152 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '409', 'content-length': '150', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-2244f3e2-8d8c-4792-8aef-959638c81197', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"conflictingRequest": {"message": "Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", "code": 409}}
19:42:37     2016-09-23 19:39:47,265 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 500 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.111s
19:42:37     2016-09-23 19:39:47,266 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '500', 'content-length': '201', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-b19a6519-ddf4-406c-b004-a544b8f4aeeb', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.InvalidInput'>", "code": 500}}
19:42:37
19:42:37
19:42:37 Captured traceback-2:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py", line 346, in delete_volume
19:42:37         cls._delete_volume(cls.volumes_extensions_client, volume_id)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py", line 277, in _delete_volume
19:42:37         volumes_client.delete_volume(volume_id)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/volumes_client.py", line 75, in delete_volume
19:42:37         resp, body = self.delete("os-volumes/%s" % volume_id)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 307, in delete
19:42:37         return self.request('DELETE', url, extra_headers, headers, body)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
19:42:37         method, url, extra_headers, headers, body, chunked)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 665, in request
19:42:37         resp, resp_body)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 829, in _error_checker
19:42:37         message=message)
19:42:37     tempest.lib.exceptions.ServerFault: Got server fault
19:42:37     Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
19:42:37     <class 'nova.exception.InvalidInput'>
19:42:37
19:42:37
19:42:37 Captured traceback-1:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py", line 73, in _detach
19:42:37         self.servers_client.detach_volume(server_id, volume_id)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 404, in detach_volume
19:42:37         (server_id, volume_id))
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 307, in delete
19:42:37         return self.request('DELETE', url, extra_headers, headers, body)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
19:42:37         method, url, extra_headers, headers, body, chunked)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 665, in request
19:42:37         resp, resp_body)
19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 778, in _error_checker
19:42:37         raise exceptions.Conflict(resp_body, resp=resp)
19:42:37     tempest.lib.exceptions.Conflict: An object with that identifier already exists
19:42:37     Details: {u'message': u"Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", u'code': 409}
19:42:37

** Affects: nova
     Importance: Undecided
         Status: Confirmed


** Tags: libvirt nfs volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1627183

Title:
  Cannot access storage file
  '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-
  67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such
  file or directory", u'code': 500

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Ran tempest, hit this error. I can't get to the system right now to
  get additional info. If i can, i will add more info.

  19:42:37
  19:42:37 ==============================
  19:42:37 Failed 1 tests - output below:
  19:42:37 ==============================
  19:42:37
  19:42:37 tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
  19:42:37 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  19:42:37
  19:42:37 Captured traceback:
  19:42:37 ~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py", line 80, in _unrescue
  19:42:37         server_id, 'ACTIVE')
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/common/waiters.py", line 75, in wait_for_server_status
  19:42:37         server_id=server_id)
  19:42:37     tempest.exceptions.BuildErrorException: Server 04e4aa2a-13ed-44ca-a275-bbb703c12493 failed to build and is in ERROR status
  19:42:37     Details: {u'message': u"Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", u'code': 500, u'created': u'2016-09-23T19:39:46Z'}
  19:42:37
  19:42:37
  19:42:37 Captured pythonlogging:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~~~
  19:42:37     2016-09-23 19:39:36,692 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes 0.411s
  19:42:37     2016-09-23 19:39:36,692 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"volume": {"display_name": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "size": 1}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes', 'x-compute-request-id': 'req-beb5ecea-c1dc-467e-9381-6c43be677b2f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.489134", "size": 1}}
  19:42:37     2016-09-23 19:39:36,817 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.122s
  19:42:37     2016-09-23 19:39:36,817 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-fa69239e-7fe6-4b8e-b3e9-786eb994a9f6', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:37,937 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.116s
  19:42:37     2016-09-23 19:39:37,938 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-b1ec1378-2e49-48cc-aaac-6f5d7218624f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:37 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "available", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:38,329 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments 0.388s
  19:42:37     2016-09-23 19:39:38,330 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': '194', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments', 'x-compute-request-id': 'req-7f079443-51fb-4f41-8797-0e542d6e2995', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
  19:42:37     2016-09-23 19:39:38,465 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.134s
  19:42:37     2016-09-23 19:39:38,466 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-586a4fca-99cf-4c35-9c7c-80b23ecf7faf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "attaching", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:39,595 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.125s
  19:42:37     2016-09-23 19:39:39,596 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '570', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-17bbc784-e95d-437c-9d20-f0a528a0762f', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "in-use", "displayDescription": null, "availabilityZone": "nova", "displayName": "tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": "67bbce20-b476-466d-b20a-8337957af780"}], "volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {"readonly": "False", "attached_mode": "rw"}, "id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:39,833 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action 0.234s
  19:42:37     2016-09-23 19:39:39,833 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"rescue": {"adminPass": "X2!oCgz87DGWc8g"}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': '32', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action', 'x-compute-request-id': 'req-e93333a1-c339-4137-8363-1ec692e93594', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"adminPass": "X2!oCgz87DGWc8g"}
  19:42:37     2016-09-23 19:39:40,069 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.234s
  19:42:37     2016-09-23 19:39:40,069 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-98335fbb-f98d-44d2-b02a-314a6687e20e', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:40 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:41,306 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.228s
  19:42:37     2016-09-23 19:39:41,306 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-fe829b17-4da3-44de-bbb9-6839c6fb4beb', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:41 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:42,541 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.227s
  19:42:37     2016-09-23 19:39:42,542 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-06e65562-36ac-440b-9b38-45eb5322fa43', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:42 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:43,799 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.243s
  19:42:37     2016-09-23 19:39:43,799 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1787', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-c693fccb-0153-4737-8192-d9869122659a', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:43 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:45,058 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.251s
  19:42:37     2016-09-23 19:39:45,058 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1767', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-be1c970e-b954-4d73-9b75-23820d0aff25', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "RESCUE", "updated": "2016-09-23T19:39:44Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:45,064 18008 INFO     [tempest.common.waiters] State transition "ACTIVE/rescuing" ==> "RESCUE/None" after 5 second wait
  19:42:37     2016-09-23 19:39:45,254 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 409 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780 0.188s
  19:42:37     2016-09-23 19:39:45,254 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '409', 'content-length': '152', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-e572880b-7dcd-457d-a06a-ed004351adaf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"conflictingRequest": {"message": "Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state rescued", "code": 409}}
  19:42:37     2016-09-23 19:39:45,396 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 202 POST https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action 0.140s
  19:42:37     2016-09-23 19:39:45,396 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"unrescue": {}}
  19:42:37         Response - Headers: {'status': '202', 'content-length': '0', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action', 'x-compute-request-id': 'req-7010ff1d-7d7c-4052-a2d5-3cddb6ddee4c', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body:
  19:42:37     2016-09-23 19:39:45,671 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.274s
  19:42:37     2016-09-23 19:39:45,672 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '1775', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-8069154d-767b-4d75-ab32-83c8d07ae745', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "unrescuing", "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "RESCUE", "updated": "2016-09-23T19:39:45Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:46,921 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 200 GET https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493 0.240s
  19:42:37     2016-09-23 19:39:46,921 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': '2010', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493', 'x-compute-request-id': 'req-99919910-98c2-40bb-90fd-0828ead5f1bf', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:46 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": "10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "self"}, {"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";, "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", "links": [{"href": "https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1";, "rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": [{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ERROR", "updated": "2016-09-23T19:39:46Z", "hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", "tenant_id": "015383e7edf74544afac10dbcea21a76", "os-extended-volumes:volumes_attached": [{"id": "67bbce20-b476-466d-b20a-8337957af780"}], "fault": {"message": "Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", "code": 500, "created": "2016-09-23T19:39:46Z"}, "config_drive": ""}}
  19:42:37     2016-09-23 19:39:46,928 18008 INFO     [tempest.common.waiters] State transition "RESCUE/unrescuing" ==> "ERROR/None" after 1 second wait
  19:42:37     2016-09-23 19:39:47,151 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 409 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780 0.222s
  19:42:37     2016-09-23 19:39:47,152 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '409', 'content-length': '150', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-2244f3e2-8d8c-4792-8aef-959638c81197', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"conflictingRequest": {"message": "Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", "code": 409}}
  19:42:37     2016-09-23 19:39:47,265 18008 INFO     [tempest.lib.common.rest_client] Request (ServerRescueNegativeTestJSON:_run_cleanups): 500 DELETE https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780 0.111s
  19:42:37     2016-09-23 19:39:47,266 18008 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '500', 'content-length': '201', 'content-location': 'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780', 'x-compute-request-id': 'req-b19a6519-ddf4-406c-b004-a544b8f4aeeb', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.InvalidInput'>", "code": 500}}
  19:42:37
  19:42:37
  19:42:37 Captured traceback-2:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py", line 346, in delete_volume
  19:42:37         cls._delete_volume(cls.volumes_extensions_client, volume_id)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py", line 277, in _delete_volume
  19:42:37         volumes_client.delete_volume(volume_id)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/volumes_client.py", line 75, in delete_volume
  19:42:37         resp, body = self.delete("os-volumes/%s" % volume_id)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 307, in delete
  19:42:37         return self.request('DELETE', url, extra_headers, headers, body)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
  19:42:37         method, url, extra_headers, headers, body, chunked)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 665, in request
  19:42:37         resp, resp_body)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 829, in _error_checker
  19:42:37         message=message)
  19:42:37     tempest.lib.exceptions.ServerFault: Got server fault
  19:42:37     Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  19:42:37     <class 'nova.exception.InvalidInput'>
  19:42:37
  19:42:37
  19:42:37 Captured traceback-1:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py", line 73, in _detach
  19:42:37         self.servers_client.detach_volume(server_id, volume_id)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 404, in detach_volume
  19:42:37         (server_id, volume_id))
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 307, in delete
  19:42:37         return self.request('DELETE', url, extra_headers, headers, body)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
  19:42:37         method, url, extra_headers, headers, body, chunked)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 665, in request
  19:42:37         resp, resp_body)
  19:42:37       File "/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 778, in _error_checker
  19:42:37         raise exceptions.Conflict(resp_body, resp=resp)
  19:42:37     tempest.lib.exceptions.Conflict: An object with that identifier already exists
  19:42:37     Details: {u'message': u"Cannot 'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", u'code': 409}
  19:42:37

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1627183/+subscriptions