yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1737599] [NEW] Instance resize with attach volume fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Samuel Matzek <smatzek@xxxxxxxxxx>
Date: Mon, 11 Dec 2017 19:02:39 -0000
Reply-to: Bug 1737599 <1737599@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

The Trove gates are failing when attempting to resize an instance what
has an ephemeral disk and an attached volume.

The stack when it fails is this:
Dec 11 03:03:28.751318 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [None req-11c69857-4556-4d83-b34c-1a0191175ceb alt_demo alt_demo] [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Setting instance vm_state to ERROR: VolumeDriverNotFound: Could not find a handler for None volume.
Dec 11 03:03:28.751537 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Traceback (most recent call last):
Dec 11 03:03:28.751683 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 7297, in _error_out_instance_on_exception
Dec 11 03:03:28.751831 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     yield
Dec 11 03:03:28.751970 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4358, in finish_resize
Dec 11 03:03:28.752120 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     disk_info, image_meta, bdms)
Dec 11 03:03:28.752261 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4326, in _finish_resize
Dec 11 03:03:28.752408 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     old_instance_type)
Dec 11 03:03:28.752545 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 11 03:03:28.752680 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self.force_reraise()
Dec 11 03:03:28.752813 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 11 03:03:28.752947 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     six.reraise(self.type_, self.value, self.tb)
Dec 11 03:03:28.753092 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4321, in _finish_resize
Dec 11 03:03:28.753229 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info, power_on)
Dec 11 03:03:28.753363 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7640, in finish_migration
Dec 11 03:03:28.753495 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info=block_device_info)
Dec 11 03:03:28.753628 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5071, in _get_guest_xml
Dec 11 03:03:28.753763 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     context)
Dec 11 03:03:28.753897 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4879, in _get_guest_config
Dec 11 03:03:28.754035 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     flavor, guest.os_type)
Dec 11 03:03:28.754173 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in _get_guest_storage_config
Dec 11 03:03:28.754303 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self._connect_volume(connection_info, info, instance)
Dec 11 03:03:28.754429 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1180, in _connect_volume
Dec 11 03:03:28.754597 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     vol_driver = self._get_volume_driver(connection_info)
Dec 11 03:03:28.754730 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1176, in _get_volume_driver
Dec 11 03:03:28.754873 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     raise exception.VolumeDriverNotFound(driver_type=driver_type)
Dec 11 03:03:28.755007 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] VolumeDriverNotFound: Could not find a handler for None volume.
Dec 11 03:03:28.755150 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] 


A link to one of the gate's Nova logs is:
http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz

** Affects: nova
     Importance: High
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737599

Title:
  Instance resize with attach volume fails

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  The Trove gates are failing when attempting to resize an instance what
  has an ephemeral disk and an attached volume.

  The stack when it fails is this:
  Dec 11 03:03:28.751318 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [None req-11c69857-4556-4d83-b34c-1a0191175ceb alt_demo alt_demo] [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Setting instance vm_state to ERROR: VolumeDriverNotFound: Could not find a handler for None volume.
  Dec 11 03:03:28.751537 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Traceback (most recent call last):
  Dec 11 03:03:28.751683 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 7297, in _error_out_instance_on_exception
  Dec 11 03:03:28.751831 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     yield
  Dec 11 03:03:28.751970 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4358, in finish_resize
  Dec 11 03:03:28.752120 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     disk_info, image_meta, bdms)
  Dec 11 03:03:28.752261 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4326, in _finish_resize
  Dec 11 03:03:28.752408 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     old_instance_type)
  Dec 11 03:03:28.752545 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Dec 11 03:03:28.752680 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self.force_reraise()
  Dec 11 03:03:28.752813 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Dec 11 03:03:28.752947 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     six.reraise(self.type_, self.value, self.tb)
  Dec 11 03:03:28.753092 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4321, in _finish_resize
  Dec 11 03:03:28.753229 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info, power_on)
  Dec 11 03:03:28.753363 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7640, in finish_migration
  Dec 11 03:03:28.753495 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info=block_device_info)
  Dec 11 03:03:28.753628 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5071, in _get_guest_xml
  Dec 11 03:03:28.753763 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     context)
  Dec 11 03:03:28.753897 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4879, in _get_guest_config
  Dec 11 03:03:28.754035 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     flavor, guest.os_type)
  Dec 11 03:03:28.754173 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in _get_guest_storage_config
  Dec 11 03:03:28.754303 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self._connect_volume(connection_info, info, instance)
  Dec 11 03:03:28.754429 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1180, in _connect_volume
  Dec 11 03:03:28.754597 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     vol_driver = self._get_volume_driver(connection_info)
  Dec 11 03:03:28.754730 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1176, in _get_volume_driver
  Dec 11 03:03:28.754873 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     raise exception.VolumeDriverNotFound(driver_type=driver_type)
  Dec 11 03:03:28.755007 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] VolumeDriverNotFound: Could not find a handler for None volume.
  Dec 11 03:03:28.755150 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] 

  
  A link to one of the gate's Nova logs is:
  http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737599/+subscriptions
Follow ups

[Bug 1737599] Re: Instance resize with new-style attach volume fails
From: OpenStack Infra, 2017-12-20