yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Changed in: tempest
     Assignee: (unassigned) => Matt Riedemann (mriedem)

** Changed in: nova
       Status: Triaged => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1722577

Title:
  test_list_get_volume_attachments failing with 400 error on teardown
  when detaching an already detached volume

Status in OpenStack Compute (nova):
  Won't Fix
Status in tempest:
  Confirmed

Bug description:
  Noticed this today:

  http://logs.openstack.org/04/510704/1/check/gate-tempest-dsvm-py35
  -ubuntu-xenial/084e7c7/console.html#_2017-10-10_07_50_05_759627

  2017-10-10 07:50:05.759627 | tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments[id-7fa563fe-f0f7-43eb-9e22-a1ece036b513]
  2017-10-10 07:50:05.759770 | ---------------------------------------------------------------------------------------------------------------------------------------------
  2017-10-10 07:50:05.759801 | 
  2017-10-10 07:50:05.759859 | Captured traceback:
  2017-10-10 07:50:05.759942 | ~~~~~~~~~~~~~~~~~~~
  2017-10-10 07:50:05.759999 |     b'Traceback (most recent call last):'
  2017-10-10 07:50:05.760082 |     b'  File "/opt/stack/new/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc'
  2017-10-10 07:50:05.760144 |     b'    return func(*args, **kwargs)'
  2017-10-10 07:50:05.760226 |     b'  File "/opt/stack/new/tempest/tempest/lib/services/compute/servers_client.py", line 441, in detach_volume'
  2017-10-10 07:50:05.760266 |     b'    (server_id, volume_id))'
  2017-10-10 07:50:05.760335 |     b'  File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 301, in delete'
  2017-10-10 07:50:05.760411 |     b"    return self.request('DELETE', url, extra_headers, headers, body)"
  2017-10-10 07:50:05.760545 |     b'  File "/opt/stack/new/tempest/tempest/lib/services/compute/base_compute_client.py", line 48, in request'
  2017-10-10 07:50:05.760602 |     b'    method, url, extra_headers, headers, body, chunked)'
  2017-10-10 07:50:05.760670 |     b'  File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 659, in request'
  2017-10-10 07:50:05.760716 |     b'    self._error_checker(resp, resp_body)'
  2017-10-10 07:50:05.760787 |     b'  File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 770, in _error_checker'
  2017-10-10 07:50:05.760852 |     b'    raise exceptions.BadRequest(resp_body, resp=resp)'
  2017-10-10 07:50:05.760903 |     b'tempest.lib.exceptions.BadRequest: Bad request'
  2017-10-10 07:50:05.761081 |     b'Details: {\'message\': "Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be \'in-use\' and attach_status must be \'attached\' to detach. (HTTP 400) (Request-ID: req-e4faf8b8-a431-4021-8480-486c94a91543)", \'code\': 400}'
  2017-10-10 07:50:05.761119 |     b''

  
  This test does the following:

  1. create a server
  2. create a volume vol1
  3. attach vol1 to the server and wait for the volume status to be in-use
  4. repeat steps 3 and 4 for vol2
  5. for each volume attachment, detach it and wait for it's status to be 'available'

  That all works. The failure is during the cleanup routines on
  teardown, they try to detach the volume but the volumes are already
  detached, which results in a 400 response from the compute API
  (because the compute API gets a 400 response from the volumev3 API):

  http://logs.openstack.org/04/510704/1/check/gate-tempest-dsvm-py35
  -ubuntu-xenial/084e7c7/console.html#_2017-10-10_07_50_05_808506

  2017-10-10 07:50:05.808506 |     b'2017-10-10 07:30:15,928 2243 INFO     [tempest.lib.common.rest_client] Request (AttachVolumeTestJSON:_run_cleanups): 400 DELETE https://15.184.65.28/compute/v2.1/servers/342d8d9e-cd9b-43af-8a25-648de6bf546e/os-volume_attachments/c1f8d572-e907-468b-b44a-60c5bee32094 0.372s'
  2017-10-10 07:50:05.808588 |     b"2017-10-10 07:30:15,928 2243 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}"
  2017-10-10 07:50:05.808610 |     b'        Body: None'
  2017-10-10 07:50:05.808826 |     b"    Response - Headers: {'content-location': 'https://15.184.65.28/compute/v2.1/servers/342d8d9e-cd9b-43af-8a25-648de6bf546e/os-volume_attachments/c1f8d572-e907-468b-b44a-60c5bee32094', 'server': 'Apache/2.4.18 (Ubuntu)', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-nova-api-version': '2.1', 'x-openstack-request-id': 'req-1a6f6f2c-b64b-4507-8b48-f9600bb4a8fd', 'openstack-api-version': 'compute 2.1', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 10 Oct 2017 07:30:15 GMT', 'connection': 'close', 'x-compute-request-id': 'req-1a6f6f2c-b64b-4507-8b48-f9600bb4a8fd', 'content-length': '269', 'status': '400'}"
  2017-10-10 07:50:05.808941 |     b'        Body: b\'{"badRequest": {"code": 400, "message": "Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be \\\'in-use\\\' and attach_status must be \\\'attached\\\' to detach. (HTTP 400) (Request-ID: req-e4faf8b8-a431-4021-8480-486c94a91543)"}}\''

  The cleanup code is handling 404s but not 400s and so the cleanup
  routine fails.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22_run_cleanups%5C%22%20AND%20message%3A%5C%22400%20DELETE%5C%22%20AND%20message%3A%5C%22compute%2Fv2.1%2Fservers%2F%5C%22%20AND%20message%3A%5C
  %22os-volume_attachments%2F%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1722577/+subscriptions