yahoo-eng-team team mailing list archive

[Solly Ross <sross@xxxxxxxxxx>]

The merged patch only fixes live migration -- nothing has changed for
cold migrations/resizes (it still just calls resize) -- I ran into this
with my work for bp use-libvirt-storage-pools.

** Changed in: nova
       Status: Fix Released => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1314526

Title:
  revert resize removes rbd shared image

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  We run multi-host nova-compute with

  libvirt_images_type=rbd
  libvirt_images_rbd_pool=compute

  Resize-confirm function works just fine.
  Resize-revert removes shared rbd for both instances image during reverting.

  Options nova.conf i've tried to change with no luck :

  allow_resize_to_same_host=True/False
  resize_fs_using_block_device=True/False
  block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC
  live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER

  Errors you can find at the bottom of the page.
  1. first error was fixed by adding  image_cache_manager_interval = 0
  2. 2nd error still active. 

  During revert process for both types of migration there is
  driver.destroy() at destination that removes original image from rbd
  storage.

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3164
  _____________
  def revert_resize(self, context, instance, migration, reservations):
             ...
             self.driver.destroy(context, instance, network_info,
                                 block_device_info)

             ...
  _____________
  that calls

  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L956
  _____________
  def destroy(self, context, instance, network_info, block_device_info=None,
               destroy_disks=True):
       self._destroy(instance)
       self.cleanup(context, instance, network_info, block_device_info,
                    destroy_disks)
  _____________

  that calls

  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1069
  _____________
  def cleanup(self, context, instance, network_info, block_device_info=None,
                  destroy_disks=True):
      ....
          if destroy_disks:
              self._delete_instance_files(instance)
              self._cleanup_lvm(instance)
              #NOTE(haomai): destroy volumes if needed
              if CONF.libvirt.images_type == 'rbd':
                  self._cleanup_rbd(instance)
      ....
  _____________

  revert_resize runs destroy function without destory_disk variable
  which makes cleanup function to delete SHARED image.

  Here is approximate solution (not a developer)

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3199

  change from :
  _____________
  self.driver.destroy(context, instance, network_info,
                                  block_device_info)
  _____________
  to:
  _____________
  destroy_disks = not (self._is_instance_storage_shared(context, instance))
               self.driver.destroy(instance, network_info,
                                  block_device_info)
                                  block_device_info, destroy_disks=destroy_disks)
  _____________

  ERROR1####################################################
  <179>Apr 28 14:14:00 [compute] node-39 <U+FEFF>nova-nova.virt.libvirt.imagebackend ERROR: error opening rbd image /var/lib/
  nova/instances/_base/103bc0322b21e499ecea1c360abc6843ab829d06
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
      read_only=read_only)
    File "/usr/lib/python2.7/dist-packages/rbd.py", line 351, in __init__
      raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
  ImageNotFound: error opening image /var/lib/nova/instances/_base/103bc0322b21e499ecea1c360abc6843ab829d06 at snapshot None
  <179>Apr 28 14:14:00 [compute] node-39 <U+FEFF>nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3160, in finish_resize
      disk_info, image)
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3128, in _finish_resize
      block_device_info, power_on)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4627, in finish_migration
      block_device_info=None, inject_files=False)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2395, in _create_image
      project_id=instance['project_id'])
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 177, in cache
      *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 638, in create_image
      self.verify_base_size(base, size)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 218, in verify_base_size
      base_size = self.get_disk_size(base)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 622, in get_disk_size
      with RBDVolumeProxy(self, name) as vol:
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
      read_only=read_only)
    File "/usr/lib/python2.7/dist-packages/rbd.py", line 351, in __init__
      raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
  ERROR2####################################################

  <179>Apr 29 08:34:13 [compute] node-39 <U+FEFF>nova-nova.virt.libvirt.driver ERROR: An error occurred while trying to launc
  h a defined domain with xml ...
  <179>Apr 29 08:34:13 [compute] node-39 <U+FEFF>nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4997, in _error_out_instance_on_exception
      yield
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2841, in finish_revert_resize
      block_device_info, power_on)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4673, in finish_revert_migration
      block_device_info, power_on)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3255, in _create_domain_and_network
      domain = self._create_domain(xml, instance=instance, power_on=power_on)
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3198, in _create_domain
      domain.XMLDesc(0))
    File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3193, in _create_domain
      domain.createWithFlags(launch_flags)
    File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
      rv = execute(f,*args,**kwargs)
    File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
      rv = meth(*args,**kwargs)
    File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
      if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/4
  kvm: -drive file=rbd:.... hiden... No such file or directory
  ####################################################

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1314526/+subscriptions