yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1360109] [NEW] nova evacuate error on boot from volume vm

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "7.NoJ" <hejiantao5@xxxxxxxxx>
Date: Fri, 22 Aug 2014 08:00:37 -0000
Reply-to: Bug 1360109 <1360109@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When I evacuate a down host vm, and if the vm is boot from volume, it is
said that there is a volume attach to it.

And when rebuilding the vm in another host, it appear this error:

"message": "internal error: process exited while connecting to monitor: W: kvm binary is deprecated, please use qemu-system-x86_64 instead
Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.

ERROR nova.virt.libvirt.driver [req-87891841-c862-4c86-b714-747d3eccdf9e 63ec1c14b8af4a9bbec4720d27717678 6865a009d773410685f94ef3896814e1] An error occurred while trying to launch a defined domain with xml: <domain type='kvm'>
  <name>instance-0000006b</name>
  <uuid>29fb8b5f-efde-4fbc-b37c-686fad6abe2d</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2013.2.4</entry>
      <entry name='serial'>40ef4edc-23b6-11e0-a12b-001e67097fc4</entry>
      <entry name='uuid'>29fb8b5f-efde-4fbc-b37c-686fad6abe2d</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='admin'>
        <secret type='ceph' uuid='09a8cb26-aa2c-416b-a5ac-7d9f188550df'/>
      </auth>

2014-08-22 14:59:16.281 21781 ERROR nova.compute.manager [req-87891841-c862-4c86-b714-747d3eccdf9e 63ec1c14b8af4a9bbec4720d27717678 6865a009d773410685f94ef3896814e1] [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Setting instance vm_state to ERROR
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Traceback (most recent call last):
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 5000, in _error_out_instance_on_exception
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     yield
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 2076, in rebuild_instance
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     block_device_info=block_device_info)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2091, in spawn
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     block_device_info, context=context)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3249, in _create_domain_and_network
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain = self._create_domain(xml, instance=instance, power_on=power_on)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3192, in _create_domain
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain.XMLDesc(0))
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3187, in _create_domain
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain.createWithFlags(launch_flags)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     rv = execute(f,*args,**kwargs)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     rv = meth(*args,**kwargs)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 728, in createWithFlags
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] libvirtError: internal error: process exited while connecting to monitor: W: kvm binary is deprecated, please use qemu-system-x86_64 instead
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] char device redirected to /dev/pts/10 (label charserial1)
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] qemu-system-x86_64: -drive file=rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:6789\;192.168.1.53\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none: error reading header from instance-0000006b_disk
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] qemu-system-x86_64: -drive file=rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:6789\;192.168.1.53\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none: could not open disk image rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:678
2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]

And then the volume can't be deleted

ERROR cinder.volume.drivers.rbd [req-77ff27fe-63ff-4e09-80fb-
ab050b440211 63ec1c14b8af4a9bbec4720d27717678
6865a009d773410685f94ef3896814e1] ImageBusy error raised while deleting
rbd volume. This may have been caused by a connection from a client that
has crashed and, if so, may be resolved by retrying the delete after 30
seconds has elapsed.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1360109

Title:
  nova evacuate error on boot from volume vm

Status in OpenStack Compute (Nova):
  New

Bug description:
  When I evacuate a down host vm, and if the vm is boot from volume, it
  is said that there is a volume attach to it.

  And when rebuilding the vm in another host, it appear this error:

  "message": "internal error: process exited while connecting to monitor: W: kvm binary is deprecated, please use qemu-system-x86_64 instead
  Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.

  ERROR nova.virt.libvirt.driver [req-87891841-c862-4c86-b714-747d3eccdf9e 63ec1c14b8af4a9bbec4720d27717678 6865a009d773410685f94ef3896814e1] An error occurred while trying to launch a defined domain with xml: <domain type='kvm'>
    <name>instance-0000006b</name>
    <uuid>29fb8b5f-efde-4fbc-b37c-686fad6abe2d</uuid>
    <memory unit='KiB'>4194304</memory>
    <currentMemory unit='KiB'>4194304</currentMemory>
    <vcpu placement='static'>2</vcpu>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>OpenStack Foundation</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>2013.2.4</entry>
        <entry name='serial'>40ef4edc-23b6-11e0-a12b-001e67097fc4</entry>
        <entry name='uuid'>29fb8b5f-efde-4fbc-b37c-686fad6abe2d</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='host-model'>
      <model fallback='allow'/>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/bin/kvm-spice</emulator>
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <auth username='admin'>
          <secret type='ceph' uuid='09a8cb26-aa2c-416b-a5ac-7d9f188550df'/>
        </auth>

  2014-08-22 14:59:16.281 21781 ERROR nova.compute.manager [req-87891841-c862-4c86-b714-747d3eccdf9e 63ec1c14b8af4a9bbec4720d27717678 6865a009d773410685f94ef3896814e1] [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Setting instance vm_state to ERROR
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Traceback (most recent call last):
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 5000, in _error_out_instance_on_exception
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     yield
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 2076, in rebuild_instance
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     block_device_info=block_device_info)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2091, in spawn
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     block_device_info, context=context)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3249, in _create_domain_and_network
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain = self._create_domain(xml, instance=instance, power_on=power_on)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3192, in _create_domain
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain.XMLDesc(0))
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3187, in _create_domain
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     domain.createWithFlags(launch_flags)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     rv = execute(f,*args,**kwargs)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     rv = meth(*args,**kwargs)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 728, in createWithFlags
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] libvirtError: internal error: process exited while connecting to monitor: W: kvm binary is deprecated, please use qemu-system-x86_64 instead
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] char device redirected to /dev/pts/10 (label charserial1)
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] qemu-system-x86_64: -drive file=rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:6789\;192.168.1.53\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none: error reading header from instance-0000006b_disk
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d] qemu-system-x86_64: -drive file=rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:6789\;192.168.1.53\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none: could not open disk image rbd:volumes/instance-0000006b_disk:id=admin:key=AQAN4dRT8LXEMBAAazDE8LV2Gmck2Ftwa5q39w==:auth_supported=cephx\;none:mon_host=192.168.1.51\:6789\;192.168.1.52\:678
  2014-08-22 14:59:16.281 21781 TRACE nova.compute.manager [instance: 29fb8b5f-efde-4fbc-b37c-686fad6abe2d]

  And then the volume can't be deleted

  ERROR cinder.volume.drivers.rbd [req-77ff27fe-63ff-4e09-80fb-
  ab050b440211 63ec1c14b8af4a9bbec4720d27717678
  6865a009d773410685f94ef3896814e1] ImageBusy error raised while
  deleting rbd volume. This may have been caused by a connection from a
  client that has crashed and, if so, may be resolved by retrying the
  delete after 30 seconds has elapsed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1360109/+subscriptions
Follow ups

[Bug 1360109] [NEW] nova evacuate error on boot from volume vm
From: 7.NoJ, 2014-08-22
References

[Bug 1360109] [NEW] nova evacuate error on boot from volume vm
From: 7.NoJ, 2014-08-22