yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

Long incomplete bug waiting for confirmation.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1276639

Title:
  block live migration does not work when a volume is attached

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Environment:
   - Two compute nodes, running Ubuntu 12.04 LTS
   - KVM Hypervisor
   - Ceph (dumpling) back-end for Cinder
   - Grizzly-level Openstack

  Steps to reproduce:
   1) Create instance and volume
   2) Attach volume to instance
   3) Attempt a block migration between compute nodes - eg: nova live-migration --block-migrate 9b85b983-dced-4574-b14c-c72e4d92982a

  Packages:
  ii  ceph                             0.67.5-1precise
  ii  ceph-common                      0.67.5-1precise
  ii  ceph-fs-common                   0.67.5-1precise
  ii  ceph-fuse                        0.67.5-1precise
  ii  ceph-mds                         0.67.5-1precise
  ii  curl                             7.29.0-1precise.ceph
  ii  kvm                              1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.13
  ii  kvm-ipxe                         1.0.0+git-3.55f6c88-0ubuntu1
  ii  libcephfs1                       0.67.5-1precise
  ii  libcurl3                         7.29.0-1precise.ceph
  ii  libcurl3-gnutls                  7.29.0-1precise.ceph
  ii  libleveldb1                      1.12.0-1precise.ceph
  ii  nova-common                      1:2013.1.4-0ubuntu1~cloud0
  ii  nova-compute                     1:2013.1.4-0ubuntu1~cloud0
  ii  nova-compute-kvm                 1:2013.1.4-0ubuntu1~cloud0
  ii  python-ceph                      0.67.5-1precise
  ii  python-cinderclient              1:1.0.3-0ubuntu1~cloud0
  ii  python-nova                      1:2013.1.4-0ubuntu1~cloud0
  ii  python-novaclient                1:2.13.0-0ubuntu1~cloud0
  ii  qemu-common                      1.0+noroms-0ubuntu14.13
  ii  qemu-kvm                         1.0+noroms-0ubuntu14.13
  ii  qemu-utils                       1.0+noroms-0ubuntu14.13
  ii  libvirt-bin                      1.0.2-0ubuntu11.13.04.5~cloud1
  ii  libvirt0                         1.0.2-0ubuntu11.13.04.5~cloud1
  ii  python-libvirt                   1.0.2-0ubuntu11.13.04.5~cloud1

  /var/log/nova/nova-compute on source:

  2014-02-05 16:36:46.014 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a
  2014-02-05 16:36:46.233 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip.
  2014-02-05 16:36:46.234 998 INFO nova.compute.manager [-] Lifecycle event 2 on VM 9b85b983-dced-4574-b14c-c72e4d92982a
  2014-02-05 16:36:46.468 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip.
  2014-02-05 16:41:09.029 998 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a
  2014-02-05 16:41:09.265 998 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During sync_power_state the instance has a pending task. Skip.
  2014-02-05 16:41:09.640 998 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Live Migration failure: Unable to read from monitor: Connection reset by peer
  2014-02-05 16:41:12.165 998 WARNING nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance shutdown by itself. Calling the stop API.
  2014-02-05 16:41:12.398 998 INFO nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Instance destroyed successfully.

  /var/log/libvirt/libvirtd.log on source:

  2014-02-05 14:41:07.607+0000: 3437: error : qemuMonitorIORead:502 : Unable to read from monitor: Connection reset by peer
  2014-02-05 14:41:09.633+0000: 3441: error : virNetClientProgramDispatchError:175 : An error occurred, but the cause is unknown
  2014-02-05 14:41:09.634+0000: 3441: error : qemuDomainObjEnterMonitorInternal:997 : operation failed: domain is no longer running
  2014-02-05 14:41:09.634+0000: 3441: warning : doPeer2PeerMigrate3:2872 : Guest instance-00000315 probably left in 'paused' state on source

  /var/log/nova/nova-compute.log on target:

  2014-02-05 16:36:38.841 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] Instance launched has CPU info:
  {"vendor": "Intel", "model": "SandyBridge", "arch": "x86_64", "features": ["pdpe1gb", "osxsave", "dca", "pcid", "pdcm", "xtpr", "tm2", "est", "smx", "vmx", "ds_cpl", "monitor", "dtes64", "pbe", "tm", "ht", "ss", "acpi", "ds", "vme"], "topology": {"cores": 6, "threads": 2, "sockets": 1}}
  2014-02-05 16:36:46.008 28458 INFO nova.compute.manager [-] Lifecycle event 0 on VM 9b85b983-dced-4574-b14c-c72e4d92982a
  2014-02-05 16:36:46.244 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005
  2014-02-05 16:41:09.634 28458 INFO nova.compute.manager [-] Lifecycle event 1 on VM 9b85b983-dced-4574-b14c-c72e4d92982a
  2014-02-05 16:41:09.899 28458 INFO nova.compute.manager [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During the sync_power process the instance has moved from host ctpcmp003 to host ctpcmp005
  2014-02-05 16:41:11.056 AUDIT nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from mountpoint /dev/vdc
  2014-02-05 16:41:11.061 WARNING nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Detaching volume from unknown instance
  2014-02-05 16:41:11.066 ERROR nova.compute.manager [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Failed to detach volume 7e2f873e-f7ba-4ee1-85ee-cee133cefdcf from /dev/vdc
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Traceback (most recent call last):
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2983, in _detach_volume
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]     mp)
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1016, in detach_volume
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]     virt_dom = self._lookup_by_name(instance_name)
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2386, in _lookup_by_name
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]     raise exception.InstanceNotFound(instance_id=instance_name)
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] InstanceNotFound: Instance instance-00000315 could not be found.
  2014-02-05 16:41:11.066 28458 TRACE nova.compute.manager [instance: 9b85b983-dced-4574-b14c-c72e4d92982a]
  2014-02-05 16:41:12.261 28458 ERROR nova.virt.libvirt.driver [-] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] During wait destroy, instance disappeared.
  2014-02-05 16:41:12.872 INFO nova.virt.libvirt.driver [req-0f0eaabf-9e29-4d45-88c9-20194be51d49 aaf3e92b69e04958b43348677ab7b38b 1859d80f51ff4180b591f7fe2668fd68] [instance: 9b85b983-dced-4574-b14c-c72e4d92982a] Deleting instance files /var/lib/nova/instances/9b85b983-dced-4574-b14c-c72e4d92982a

  /etc/nova/nova.conf includes the directive: vncserver_listen=0.0.0.0

  root@ctpcmp003:~# grep "tls\|tcp" /etc/libvirt/libvirtd.conf | grep -v "^#"
  listen_tls = 0
  listen_tcp = 1
  auth_tcp = "none"

  root@ctpcmp003:~# grep vncserver_listen /etc/nova/nova.conf
  vncserver_listen=0.0.0.0

  root@ctpcmp003:~# ps -ef | grep libvirtd
  root     20608     1  0 16:29 ?        00:00:11 /usr/sbin/libvirtd -d -l

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1276639/+subscriptions