yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1476806] Re: Unable to delete instance with attached volumes which failed to boot

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Mon, 03 Aug 2015 21:21:37 -0000
Reply-to: Bug 1476806 <1476806@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
@Matthew, this isn't a great solution but you could delete the related
block_device_mapping records for the instance from the nova database.

** Also affects: python-cinderclient
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1476806

Title:
  Unable to delete instance with attached volumes which failed to boot

Status in OpenStack Compute (nova):
  In Progress
Status in python-cinderclient:
  New

Bug description:
  I ran devstack deployment on this git nova version:

  commit 35375133398d862a61334783c1e7a90b95f34cdb
  Merge: 83623dd b2c5542
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Thu Jul 16 02:01:05 2015 +0000

      Merge "Port crypto to Python 3"

  If you try to start an instance with the following config and end up
  with the following erro:

   Error defining a domain with XML: <domain type="parallels">
    <uuid>f81e862a-644b-4145-ab44-86d5c468106f</uuid>
    <name>instance-00000001</name>
    <memory>2097152</memory>
    <vcpu>1</vcpu>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="12.0.0"/>
        <nova:name>ct-volume</nova:name>
        <nova:creationTime>2015-07-21 17:46:34</nova:creationTime>
        <nova:flavor name="m1.small">
          <nova:memory>2048</nova:memory>
          <nova:disk>20</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>1</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="5ff3594c1b8b4694acf2cf2ee13a27ac">admin</nova:user>
          <nova:project uuid="ee1f664443ef4f1e8056b45baa1e83a5">demo</nova:project>
        </nova:owner>
      </nova:instance>
    </metadata>
    <os>
      <type>hvm</type>
      <boot dev="hd"/>
    </os>
    <clock offset="utc"/>
    <devices>
      <disk type="block" device="disk">
        <driver type="raw" cache="none"/>
        <source dev="/dev/disk/by-path/ip-10.27.68.210:3260-iscsi-iqn.2010-10.org.openstack:volume-b147e00f-000f-4fbc-8141-afeb44e92549-lun-1"/>
        <target bus="sata" dev="sda"/>
        <serial>b147e00f-000f-4fbc-8141-afeb44e92549</serial>
      </disk>
      <interface type="bridge">
        <mac address="fa:16:3e:3f:f4:1a"/>
        <source bridge="qbr5a84792b-d8"/>
        <target dev="tap5a84792b-d8"/>
      </interface>
      <graphics type="vnc" autoport="yes" listen="10.27.68.210"/>
      <video>
        <model type="vga"/>
      </video>
    </devices>
  </domain>

  Then you can't terminate the instance with the following error:

  2015-07-21 13:54:15.418 ERROR nova.compute.manager [req-8184e7f2-5cec-4c51-9f24-f39a17d8b6eb admin demo] [instance: f81e862a-644b-4145-ab44-86d5c468106f] Setting instance vm_state to ERROR
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f] Traceback (most recent call last):
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/compute/manager.py", line 2361, in do_terminate_instance
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     self._delete_instance(context, instance, bdms, quotas)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/hooks.py", line 149, in inner
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     rv = f(*args, **kwargs)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/compute/manager.py", line 2340, in _delete_instance
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     quotas.rollback()
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/compute/manager.py", line 2310, in _delete_instance
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     self._shutdown_instance(context, instance, bdms)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/compute/manager.py", line 2246, in _shutdown_instance
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     self.volume_api.detach(context, bdm.volume_id)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/volume/cinder.py", line 224, in wrapper
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     six.reraise(exc_value, None, exc_trace)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/volume/cinder.py", line 213, in wrapper
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/vz/stack/nova/nova/volume/cinder.py", line 369, in detach
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     cinderclient(context).volumes.detach(volume_id)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 407, in detach
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     {'attachment_id': attachment_uuid})
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 379, in _action
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     return self.api.client.post(url, body=body)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 104, in post
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     return self._cs_request(url, 'POST', **kwargs)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 98, in _cs_request
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     return self.request(url, method, **kwargs)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 91, in request
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f]     raise exceptions.from_response(resp, body)
  2015-07-21 13:54:15.418 17566 ERROR nova.compute.manager [instance: f81e862a-644b-4145-ab44-86d5c468106f] InvalidInput: Invalid input received: Error detaching volume - InvalidVolume: Invalid volume: Detach volume failed, because there are currently no active attachments. (HTTP 400) (Request-ID: req-a38c7fe4-058a-4288-b2f8-a4c75838abd9)


  2015-07-21 13:54:15.679 ERROR oslo_messaging.rpc.dispatcher [req-8184e7f2-5cec-4c51-9f24-f39a17d8b6eb admin demo] Exception during message handling: Invalid input received: Error detaching volume - InvalidVolume: Invalid volume: Detach volume failed, because there are currently no active attachments. (HTTP 400) (Request-ID: req-a38c7fe4-058a-4288-b2f8-a4c75838abd9)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/exception.py", line 89, in wrapped
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     payload)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/exception.py", line 72, in wrapped
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 335, in decorated_function
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 306, in decorated_function
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 385, in decorated_function
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 363, in decorated_function
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 351, in decorated_function
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2373, in terminate_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     do_terminate_instance(instance, bdms)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 252, in inner
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2371, in do_terminate_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     self._set_instance_error_state(context, instance)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2361, in do_terminate_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     self._delete_instance(context, instance, bdms, quotas)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/hooks.py", line 149, in inner
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     rv = f(*args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2340, in _delete_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     quotas.rollback()
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2310, in _delete_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     self._shutdown_instance(context, instance, bdms)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/compute/manager.py", line 2246, in _shutdown_instance
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     self.volume_api.detach(context, bdm.volume_id)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/volume/cinder.py", line 224, in wrapper
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     six.reraise(exc_value, None, exc_trace)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/volume/cinder.py", line 213, in wrapper
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/vz/stack/nova/nova/volume/cinder.py", line 369, in detach
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     cinderclient(context).volumes.detach(volume_id)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 407, in detach
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     {'attachment_id': attachment_uuid})
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 379, in _action
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return self.api.client.post(url, body=body)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 104, in post
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return self._cs_request(url, 'POST', **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 98, in _cs_request
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     return self.request(url, method, **kwargs)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 91, in request
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher     raise exceptions.from_response(resp, body)
  2015-07-21 13:54:15.679 17566 ERROR oslo_messaging.rpc.dispatcher InvalidInput: Invalid input received: Error detaching volume - InvalidVolume: Invalid volume: Detach volume failed, because there are currently no active attachments. (HTTP 400) (Request-ID: req-a38c7fe4-058a-4288-b2f8-a4c75838abd9)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1476806/+subscriptions
References

[Bug 1476806] [NEW] Unable to delete instance with attached volumes which failed to boot
From: Maxim Nestratov, 2015-07-21