yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1606136] [NEW] cannot detach_volume while VM is in error state

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Silvan Kaiser <silvan@xxxxxxxxxxx>
Date: Mon, 25 Jul 2016 07:35:09 -0000
Reply-to: Bug 1606136 <1606136@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This Error occurs randomly with the Quobyte CI on arbitrary changes.
Nova tries to to detach a volume from a VM that is in vm_state error.

CI run examples can be found at [1][2]

Example output:
==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/servers/test_server_rescue_negative.py", line 80, in _unrescue
        server_id, 'ACTIVE')
      File "tempest/common/waiters.py", line 77, in wait_for_server_status
        server_id=server_id)
    tempest.exceptions.BuildErrorException: Server 35a2d53a-d80d-46b2-87a6-a7a82f9d5244 failed to build and is in ERROR status
    Details: {u'code': 500, u'message': u"Failed to open file '/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-351db2c5-9724-410f-b1d8-8680065c0788': No such file or directory", u'created': u'2016-07-23T13:03:20Z'}
    

Captured traceback-2:
~~~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/base.py", line 346, in delete_volume
        cls._delete_volume(cls.volumes_extensions_client, volume_id)
      File "tempest/api/compute/base.py", line 277, in _delete_volume
        volumes_client.delete_volume(volume_id)
      File "tempest/lib/services/compute/volumes_client.py", line 63, in delete_volume
        resp, body = self.delete("os-volumes/%s" % volume_id)
      File "tempest/lib/common/rest_client.py", line 301, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
        method, url, extra_headers, headers, body, chunked)
      File "tempest/lib/common/rest_client.py", line 664, in request
        resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 828, in _error_checker
        message=message)
    tempest.lib.exceptions.ServerFault: Got server fault
    Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
    <class 'nova.exception.InvalidInput'>
    

Captured traceback-1:
~~~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/servers/test_server_rescue_negative.py", line 73, in _detach
        self.servers_client.detach_volume(server_id, volume_id)
      File "tempest/lib/services/compute/servers_client.py", line 342, in detach_volume
        (server_id, volume_id))
      File "tempest/lib/common/rest_client.py", line 301, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
        method, url, extra_headers, headers, body, chunked)
      File "tempest/lib/common/rest_client.py", line 664, in request
        resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 777, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: An object with that identifier already exists
    Details: {u'code': 409, u'message': u"Cannot 'detach_volume' instance 35a2d53a-d80d-46b2-87a6-a7a82f9d5244 while it is in vm_state error"}


[1] http://78.46.57.153:8081/refs-changes-38-346438-3/
[2] http://78.46.57.153:8081/refs-changes-58-346358-1/

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1606136

Title:
  cannot detach_volume while VM is in error state

Status in OpenStack Compute (nova):
  New

Bug description:
  This Error occurs randomly with the Quobyte CI on arbitrary changes.
  Nova tries to to detach a volume from a VM that is in vm_state error.

  CI run examples can be found at [1][2]

  Example output:
  ==============================
  Failed 1 tests - output below:
  ==============================

  tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/api/compute/servers/test_server_rescue_negative.py", line 80, in _unrescue
          server_id, 'ACTIVE')
        File "tempest/common/waiters.py", line 77, in wait_for_server_status
          server_id=server_id)
      tempest.exceptions.BuildErrorException: Server 35a2d53a-d80d-46b2-87a6-a7a82f9d5244 failed to build and is in ERROR status
      Details: {u'code': 500, u'message': u"Failed to open file '/mnt/quobyte-volume/abfa1002557ab2b21ec218a86487dd92/volume-351db2c5-9724-410f-b1d8-8680065c0788': No such file or directory", u'created': u'2016-07-23T13:03:20Z'}
      

  Captured traceback-2:
  ~~~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/api/compute/base.py", line 346, in delete_volume
          cls._delete_volume(cls.volumes_extensions_client, volume_id)
        File "tempest/api/compute/base.py", line 277, in _delete_volume
          volumes_client.delete_volume(volume_id)
        File "tempest/lib/services/compute/volumes_client.py", line 63, in delete_volume
          resp, body = self.delete("os-volumes/%s" % volume_id)
        File "tempest/lib/common/rest_client.py", line 301, in delete
          return self.request('DELETE', url, extra_headers, headers, body)
        File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
          method, url, extra_headers, headers, body, chunked)
        File "tempest/lib/common/rest_client.py", line 664, in request
          resp, resp_body)
        File "tempest/lib/common/rest_client.py", line 828, in _error_checker
          message=message)
      tempest.lib.exceptions.ServerFault: Got server fault
      Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
      <class 'nova.exception.InvalidInput'>
      

  Captured traceback-1:
  ~~~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/api/compute/servers/test_server_rescue_negative.py", line 73, in _detach
          self.servers_client.detach_volume(server_id, volume_id)
        File "tempest/lib/services/compute/servers_client.py", line 342, in detach_volume
          (server_id, volume_id))
        File "tempest/lib/common/rest_client.py", line 301, in delete
          return self.request('DELETE', url, extra_headers, headers, body)
        File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
          method, url, extra_headers, headers, body, chunked)
        File "tempest/lib/common/rest_client.py", line 664, in request
          resp, resp_body)
        File "tempest/lib/common/rest_client.py", line 777, in _error_checker
          raise exceptions.Conflict(resp_body, resp=resp)
      tempest.lib.exceptions.Conflict: An object with that identifier already exists
      Details: {u'code': 409, u'message': u"Cannot 'detach_volume' instance 35a2d53a-d80d-46b2-87a6-a7a82f9d5244 while it is in vm_state error"}

  
  [1] http://78.46.57.153:8081/refs-changes-38-346438-3/
  [2] http://78.46.57.153:8081/refs-changes-58-346358-1/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1606136/+subscriptions
Follow ups

[Bug 1606136] Re: Fail to unrescue server with quobyte volume attached
From: Silvan Kaiser, 2017-01-20