yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #30984
[Bug 1157725] Re: Snapshot stuck in 'queued' for VM after Volume detach
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1157725
Title:
Snapshot stuck in 'queued' for VM after Volume detach
Status in OpenStack Compute (Nova):
Expired
Bug description:
Problem statement:
A Vm can't have a snapshot taken successfully once it has had a volume attached and detached.
Current versions -
Current version of OS - Linux 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Current version of Nova-compute - 2012.2.1+stable-20121212-a99a802e-0ubuntu1.2~cloud0
Current version of CEPH - 0.56.3-42-ga30903c-1precise
Consistency of error - 100%
No. of Openstack instances tested on - 2
Steps to reproduce:
Create a VM via the horizon dashboard
Create a volume with cinder - any size (10gb used for testing)
Attach the volume to the Created VM
format/use the volume and then unmount
detach the volume from the created VM
click on create snapshot , name and click on create snapshot
The Snapshot will remain in the queued state.
Vm reboot actions fail to start the VM - it is shutdown and offline
The Vm is shutdown and cant be started. If you attempt to start the VM via virsh you get this error message
virsh start instance-0000001a
error: Failed to start domain instance-0000001a
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/5
Unknown savevm section or instance '0000:00:06.0/virtio-blk' 0
load of migration failed
the only way to restore the Vm to service is to dump the saved state from virsh as follows:
virsh managedsave-remove NameOfDomain
The Vm will the able to be restarted
workarounds identified:
Not a real workaround but brings the Vm back online:
ID the host running the VM, log onto that host
virsh managedsave-remove <NameOfDomain> to dump the failed suspend state data
virsh start <NameOfDomain> OR nova reboot <Name of VM>
Vm should then restart
error messages produced:
Nova-compute.log
2013-03-20 13:22:29 AUDIT nova.compute.manager [req-4f9d1cc7-c476-4add-9cc2-e58801d752c1 e560e0362abe492db88c655dc043ce4f cbdfbd6c58d444fcb858e77fc7135f04] [instance: 7a0df355-6094-4d2d-8ca9-f2453222
76e5] instance snapshotting
2013-03-20 13:23:06 1939 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 175, in decorated_function
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp pass
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 161, in decorated_function
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 196, in decorated_function
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 190, in decorated_function
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1174, in snapshot_instance
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self.driver.snapshot(context, instance, image_id)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 875, in snapshot
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp self._create_domain(domain=virt_dom)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1910, in _create_domain
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp domain.createWithFlags(launch_flags)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 650, in createWithFlags
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/5
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp Unknown savevm section or instance '0000:00:06.0/virtio-blk' 0
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp load of migration failed
2013-03-20 13:23:06 1939 TRACE nova.openstack.common.rpc.amqp 2013-03-20 13:23:03.154+0000: shutting down
Libvirt instance log
2013-03-20 13:23:00.091+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name instance-0000001a -S -M pc-1.0 -cpu core2duo,+lahf_lm,+rdtscp,+pdpe1gb,+popcnt,+sse4.2,+sse4.1,+dca,+pdcm,+xtpr,+cx16,+tm2,+est,+smx,+vmx,+ds_cpl,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid 7a0df355-6094-4d2d-8ca9-f245322276e5 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000001a.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/instance-0000001a/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:48:bc:cc,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/instance-0000001a/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 172.28.107.199:3 -k en-us -vga cirrus -incoming fd:25 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5
Unknown savevm section or instance '0000:00:06.0/virtio-blk' 0
load of migration failed
2013-03-20 13:23:03.154+0000: shutting down
Any help or guidance on how to resolve or fully work around this error would be really appreciated
thanks
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1157725/+subscriptions
