← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1397893] Re: Undeletable volume-backed instance

 

This was fixed in juno here: https://review.openstack.org/#/c/104048/

** Changed in: nova
       Status: Incomplete => Fix Committed

** Changed in: nova
     Assignee: (unassigned) => Roman Podoliaka (rpodolyaka)

** Changed in: nova
     Assignee: Roman Podoliaka (rpodolyaka) => Vladik Romanovsky (vladik-romanovsky)

** Tags added: in-stable-juno

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1397893

Title:
  Undeletable volume-backed instance

Status in OpenStack Compute (nova):
  Fix Released
Status in nova package in Ubuntu:
  In Progress

Bug description:
  nova-compute package version 1:2014.1.3-0ubuntu1.1 on Ubuntu 14.04.

  Trying to delete a volume-backed instance in error state doesn't work.
  Nova-compute logs the following error.

  # nova delete 71a41e09-e8bc-4829-979e-1d175246da00
  # tail /var/log/nova/nova-compute.log
  [...]
  2014-12-01 09:52:25.563 25832 AUDIT nova.compute.manager [req-39acd9da-518c-4804-bf30-1a38eace21bf 4474a81aca524682875658eb8064c33d 7dbed2bcbd7541289c34ae8392acf612] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Terminating 
  instance
  2014-12-01 09:52:25.569 25832 ERROR nova.virt.libvirt.driver [-] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] During wait destroy, instance disappeared.
  2014-12-01 09:52:25.638 25832 ERROR nova.compute.manager [req-39acd9da-518c-4804-bf30-1a38eace21bf 4474a81aca524682875658eb8064c33d 7dbed2bcbd7541289c34ae8392acf612] [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Setting inst
  ance vm_state to ERROR
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] Traceback (most recent call last):
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2273, in do_terminate_instance
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     self._delete_instance(context, instance, bdms, quotas)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 103, in inner
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     rv = f(*args, **kwargs)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2242, in _delete_instance
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     quotas.rollback()
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     six.reraise(self.type_, self.value, self.tb)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2214, in _delete_instance
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     self._shutdown_instance(context, db_inst, bdms)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2156, in _shutdown_instance
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     requested_networks)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     six.reraise(self.type_, self.value, self.tb)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2146, in _shutdown_instance
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     block_device_info)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 963, in destroy
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     destroy_disks)
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1040, in cleanup
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00]     disk_dev = vol['mount_device'].rpartition("/")[2]
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] AttributeError: 'NoneType' object has no attribute 'rpartition'
  2014-12-01 09:52:25.638 25832 TRACE nova.compute.manager [instance: 71a41e09-e8bc-4829-979e-1d175246da00] 
  [...]

  # nova show 71a41e09-e8bc-4829-979e-1d175246da00
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Property                             | Value                                                                                                                                                                                        |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                                       |
  | OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                         |
  | OS-EXT-SRV-ATTR:host                 | openstack110                                                                                                                                                                                 |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | openstack110.local                                                                                                                                                          |
  | OS-EXT-SRV-ATTR:instance_name        | instance-00000b40                                                                                                                                                                            |
  | OS-EXT-STS:power_state               | 0                                                                                                                                                                                            |
  | OS-EXT-STS:task_state                | -                                                                                                                                                                                            |
  | OS-EXT-STS:vm_state                  | error                                                                                                                                                                                        |
  | OS-SRV-USG:launched_at               | -                                                                                                                                                                                            |
  | OS-SRV-USG:terminated_at             | -                                                                                                                                                                                            |
  | accessIPv4                           |                                                                                                                                                                                              |
  | accessIPv6                           |                                                                                                                                                                                              |
  | config_drive                         |                                                                                                                                                                                              |
  | created                              | 2014-10-13T14:14:54Z                                                                                                                                                                         |
  | fault                                | {"message": "'NoneType' object has no attribute 'rpartition'", "code": 500, "details": "  File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 290, in decorated_function |
  |                                      |     return function(self, context, *args, **kwargs)                                                                                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2285, in terminate_instance                                                                                        |
  |                                      |     do_terminate_instance(instance, bdms)                                                                                                                                                    |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 249, in inner                                                                                           |
  |                                      |     return f(*args, **kwargs)                                                                                                                                                                |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2283, in do_terminate_instance                                                                                     |
  |                                      |     self._set_instance_error_state(context, instance['uuid'])                                                                                                                                |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                                                                          |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                                                                             |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2273, in do_terminate_instance                                                                                     |
  |                                      |     self._delete_instance(context, instance, bdms, quotas)                                                                                                                                   |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 103, in inner                                                                                                                |
  |                                      |     rv = f(*args, **kwargs)                                                                                                                                                                  |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2242, in _delete_instance                                                                                          |
  |                                      |     quotas.rollback()                                                                                                                                                                        |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                                                                          |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                                                                             |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2214, in _delete_instance                                                                                          |
  |                                      |     self._shutdown_instance(context, db_inst, bdms)                                                                                                                                          |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2156, in _shutdown_instance                                                                                        |
  |                                      |     requested_networks)                                                                                                                                                                      |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 68, in __exit__                                                                                          |
  |                                      |     six.reraise(self.type_, self.value, self.tb)                                                                                                                                             |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2146, in _shutdown_instance                                                                                        |
  |                                      |     block_device_info)                                                                                                                                                                       |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 963, in destroy                                                                                                |
  |                                      |     destroy_disks)                                                                                                                                                                           |
  |                                      |   File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 1040, in cleanup                                                                                               |
  |                                      |     disk_dev = vol['mount_device'].rpartition(\"/\")[2]                                                                                                                                      |
  |                                      | ", "created": "2014-12-01T08:52:25Z"}                                                                                                                                                        |
  | flavor                               | m1.large (4)                                                                                                                                                                                 |
  | hostId                               | 567b019ea0119d6b7c09dcf0d8d859ad64a8cca16254148f439cd8b6                                                                                                                                     |
  | id                                   | 71a41e09-e8bc-4829-979e-1d175246da00                                                                                                                                                         |
  | image                                | openstack100-clone-1 (b472f236-ec54-41a3-9e92-40f091d57d05)                                                                                                                                  |
  | key_name                             | francois                                                                                                                                                                                     |
  | metadata                             | {}                                                                                                                                                                                           |
  | name                                 | openstack100-clone                                                                                                                                                                           |
  | os-extended-volumes:volumes_attached | [{"id": "d8d89370-f334-4f78-bb35-e4a74f60d077"}]                                                                                                                                             |
  | status                               | ERROR                                                                                                                                                                                        |
  | tenant_id                            | 7dbed2bcbd7541289c34ae8392acf612                                                                                                                                                             |
  | updated                              | 2014-12-01T08:52:25Z                                                                                                                                                                         |
  | user_id                              | 4474a81aca524682875658eb8064c33d                                                                                                                                                             |
  +--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  # cinder show d8d89370-f334-4f78-bb35-e4a74f60d077
  +--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  |            Property            |                                                                                                                                  Value                                                                                                                                  |
  +--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  |          attachments           |                                           [{u'id': u'd8d89370-f334-4f78-bb35-e4a74f60d077', u'server_id': u'71a41e09-e8bc-4829-979e-1d175246da00', u'host_name': None, u'volume_id': u'd8d89370-f334-4f78-bb35-e4a74f60d077'}]                                          |
  |       availability_zone        |                                                                                                                                   nova                                                                                                                                  |
  |            bootable            |                                                                                                                                   true                                                                                                                                  |
  |           created_at           |                                                                                                                        2014-10-13T14:14:55.000000                                                                                                                       |
  |      display_description       |                                                                                                                                                                                                                                                                         |
  |          display_name          |                                                                                                                                                                                                                                                                         |
  |           encrypted            |                                                                                                                                  False                                                                                                                                  |
  |               id               |                                                                                                                   d8d89370-f334-4f78-bb35-e4a74f60d077                                                                                                                  |
  |            metadata            |                                                                                                             {u'readonly': u'False', u'attached_mode': u'rw'}                                                                                                            |
  |     os-vol-host-attr:host      |                                                                                                                               openstack100                                                                                                                              |
  | os-vol-mig-status-attr:migstat |                                                                                                                                   None                                                                                                                                  |
  | os-vol-mig-status-attr:name_id |                                                                                                                                   None                                                                                                                                  |
  |  os-vol-tenant-attr:tenant_id  |                                                                                                                     7dbed2bcbd7541289c34ae8392acf612                                                                                                                    |
  |              size              |                                                                                                                                   100                                                                                                                                   |
  |          snapshot_id           |                                                                                                                                   None                                                                                                                                  |
  |          source_volid          |                                                                                                                                   None                                                                                                                                  |
  |             status             |                                                                                                                                available                                                                                                                                |
  |     volume_image_metadata      | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'openstack100-clone-1', u'image_id': u'b472f236-ec54-41a3-9e92-40f091d57d05', u'checksum': u'e3898f7109a6b496b39245988bbdd1e0', u'min_disk': u'80', u'size': u'68893671424'} |
  |          volume_type           |                                                                                                                                   None                                                                                                                                  |
  +--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1397893/+subscriptions