yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #45411
[Bug 1314526] Re: revert resize removes rbd shared image
*** This bug is a duplicate of bug 1399244 ***
https://bugs.launchpad.net/bugs/1399244
Reviewed: https://review.openstack.org/187395
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=29476a67d4d4ff7eedd97f589422d47015111448
Submitter: Jenkins
Branch: master
commit 29476a67d4d4ff7eedd97f589422d47015111448
Author: Nicolas Simonds <nic@xxxxxxxxxxxxx>
Date: Mon Jun 1 13:58:37 2015 -0700
libvirt: Fix/implement revert-resize for RBD-backed images
* Makes a snapshot of Ceph-backed roots prior to resize
* Rolls back to snapshot on revert
* Destroys resize snapshots on image cleanup
Closes-Bug: 1369465
Closes-Bug: 1314526
Change-Id: I328d2c41696a9c0f090f822a51ea42fac83f62ec
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1314526
Title:
revert resize removes rbd shared image
Status in OpenStack Compute (nova):
Fix Released
Bug description:
We run multi-host nova-compute with
libvirt_images_type=rbd
libvirt_images_rbd_pool=compute
Resize-confirm function works just fine.
Resize-revert removes shared rbd for both instances image during reverting.
Options nova.conf i've tried to change with no luck :
allow_resize_to_same_host=True/False
resize_fs_using_block_device=True/False
block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER
Errors you can find at the bottom of the page.
1. first error was fixed by adding image_cache_manager_interval = 0
2. 2nd error still active.
During revert process for both types of migration there is
driver.destroy() at destination that removes original image from rbd
storage.
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3164
_____________
def revert_resize(self, context, instance, migration, reservations):
...
self.driver.destroy(context, instance, network_info,
block_device_info)
...
_____________
that calls
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L956
_____________
def destroy(self, context, instance, network_info, block_device_info=None,
destroy_disks=True):
self._destroy(instance)
self.cleanup(context, instance, network_info, block_device_info,
destroy_disks)
_____________
that calls
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1069
_____________
def cleanup(self, context, instance, network_info, block_device_info=None,
destroy_disks=True):
....
if destroy_disks:
self._delete_instance_files(instance)
self._cleanup_lvm(instance)
#NOTE(haomai): destroy volumes if needed
if CONF.libvirt.images_type == 'rbd':
self._cleanup_rbd(instance)
....
_____________
revert_resize runs destroy function without destory_disk variable
which makes cleanup function to delete SHARED image.
Here is approximate solution (not a developer)
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3199
change from :
_____________
self.driver.destroy(context, instance, network_info,
block_device_info)
_____________
to:
_____________
destroy_disks = not (self._is_instance_storage_shared(context, instance))
self.driver.destroy(instance, network_info,
block_device_info)
block_device_info, destroy_disks=destroy_disks)
_____________
ERROR1####################################################
<179>Apr 28 14:14:00 [compute] node-39 <U+FEFF>nova-nova.virt.libvirt.imagebackend ERROR: error opening rbd image /var/lib/
nova/instances/_base/103bc0322b21e499ecea1c360abc6843ab829d06
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
read_only=read_only)
File "/usr/lib/python2.7/dist-packages/rbd.py", line 351, in __init__
raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
ImageNotFound: error opening image /var/lib/nova/instances/_base/103bc0322b21e499ecea1c360abc6843ab829d06 at snapshot None
<179>Apr 28 14:14:00 [compute] node-39 <U+FEFF>nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3160, in finish_resize
disk_info, image)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3128, in _finish_resize
block_device_info, power_on)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4627, in finish_migration
block_device_info=None, inject_files=False)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2395, in _create_image
project_id=instance['project_id'])
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 177, in cache
*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 638, in create_image
self.verify_base_size(base, size)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 218, in verify_base_size
base_size = self.get_disk_size(base)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 622, in get_disk_size
with RBDVolumeProxy(self, name) as vol:
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
read_only=read_only)
File "/usr/lib/python2.7/dist-packages/rbd.py", line 351, in __init__
raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
ERROR2####################################################
<179>Apr 29 08:34:13 [compute] node-39 <U+FEFF>nova-nova.virt.libvirt.driver ERROR: An error occurred while trying to launc
h a defined domain with xml ...
<179>Apr 29 08:34:13 [compute] node-39 <U+FEFF>nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4997, in _error_out_instance_on_exception
yield
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2841, in finish_revert_resize
block_device_info, power_on)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4673, in finish_revert_migration
block_device_info, power_on)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3255, in _create_domain_and_network
domain = self._create_domain(xml, instance=instance, power_on=power_on)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3198, in _create_domain
domain.XMLDesc(0))
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3193, in _create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
rv = execute(f,*args,**kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
rv = meth(*args,**kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/4
kvm: -drive file=rbd:.... hiden... No such file or directory
####################################################
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1314526/+subscriptions
References
