yahoo-eng-team team mailing list archive

[OpenStack Infra <1737599@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/527228
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4f61f9be1704d3ed51249a10360b8c48e4cd53ff
Submitter: Zuul
Branch:    master

commit 4f61f9be1704d3ed51249a10360b8c48e4cd53ff
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Mon Dec 11 15:33:43 2017 -0500

    Update and complete volume attachments during resize
    
    With the new cinder volume attachment flow, during a resize the
    source compute will create a new volume attachment for any BDMs
    connected to the instance and delete the existing volume attachments
    which represents the connection to the source host. The finish_resize
    flow on the destination compute will then refresh the connection info
    on the BDMs it's working with before passing them to the virt driver.
    
    Since the source compute updated the BDM.attachment_id to point at
    the new attachment which is meant for the destination host, the
    refresh_connection_info call will get the connection_info for the new
    reserved attachment, which is actually empty since it hasn't been
    connected to the destination host yet. This results in wiping out the
    BDM.connection_info which has the "driver_volume_type" value which is
    something the virt driver on the destination compute needs to know
    which volume backend driver to use to connect the volume to the
    destination host.
    
    This change updates the volume attachments on the destination host
    before refreshing the connection_info in the BDMs and before we call
    the driver to finish the resize, and then we also complete the
    volume attachments once the driver has successfully finished the
    resize. This is similar to what would happen via _prep_block_device
    during the initial instance create, but since the code paths into the
    driver are different we have to handle this explicitly in the compute
    manager.
    
    Similarly, we have to perform the same incantation when reverting a
    resize and going back to the original source host.
    
    Closes-Bug: #1737599
    
    Change-Id: Ifc80d07d94311534fd9e7824ede9d09223a011c2


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737599

Title:
  Instance resize with new-style attach volume fails

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The Trove gates are failing when attempting to resize an instance what
  has an ephemeral disk and an attached volume.

  The stack when it fails is this:
  Dec 11 03:03:28.751318 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [None req-11c69857-4556-4d83-b34c-1a0191175ceb alt_demo alt_demo] [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Setting instance vm_state to ERROR: VolumeDriverNotFound: Could not find a handler for None volume.
  Dec 11 03:03:28.751537 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] Traceback (most recent call last):
  Dec 11 03:03:28.751683 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 7297, in _error_out_instance_on_exception
  Dec 11 03:03:28.751831 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     yield
  Dec 11 03:03:28.751970 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4358, in finish_resize
  Dec 11 03:03:28.752120 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     disk_info, image_meta, bdms)
  Dec 11 03:03:28.752261 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4326, in _finish_resize
  Dec 11 03:03:28.752408 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     old_instance_type)
  Dec 11 03:03:28.752545 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Dec 11 03:03:28.752680 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self.force_reraise()
  Dec 11 03:03:28.752813 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Dec 11 03:03:28.752947 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     six.reraise(self.type_, self.value, self.tb)
  Dec 11 03:03:28.753092 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4321, in _finish_resize
  Dec 11 03:03:28.753229 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info, power_on)
  Dec 11 03:03:28.753363 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7640, in finish_migration
  Dec 11 03:03:28.753495 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     block_device_info=block_device_info)
  Dec 11 03:03:28.753628 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5071, in _get_guest_xml
  Dec 11 03:03:28.753763 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     context)
  Dec 11 03:03:28.753897 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4879, in _get_guest_config
  Dec 11 03:03:28.754035 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     flavor, guest.os_type)
  Dec 11 03:03:28.754173 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in _get_guest_storage_config
  Dec 11 03:03:28.754303 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     self._connect_volume(connection_info, info, instance)
  Dec 11 03:03:28.754429 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1180, in _connect_volume
  Dec 11 03:03:28.754597 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     vol_driver = self._get_volume_driver(connection_info)
  Dec 11 03:03:28.754730 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1176, in _get_volume_driver
  Dec 11 03:03:28.754873 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7]     raise exception.VolumeDriverNotFound(driver_type=driver_type)
  Dec 11 03:03:28.755007 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] VolumeDriverNotFound: Could not find a handler for None volume.
  Dec 11 03:03:28.755150 ubuntu-xenial-rax-dfw-0001351106 nova-compute[28059]: ERROR nova.compute.manager [instance: 85cdb482-63a5-487a-b103-95b9383ffcc7] 

  
  A link to one of the gate's Nova logs is:
  http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737599/+subscriptions