yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1312874] Re: resizing an instance - causes the drives to disappear - With it hung during reboot

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 17 Sep 2014 13:56:11 -0000
Reply-to: Bug 1312874 <1312874@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1312874

Title:
  resizing an instance - causes the drives to disappear  -  With it hung
  during reboot

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  I have my openstack cluster on Havvana.

  Resizing an instance (Deb-7-based) to have more disk space - followed
  by a reboot (soft) - causes the drives to dissapear. And the instance
  would be hung permanently at the boot screen for it cannot find any
  drives.

  STEPS:
  ----------
  1. Create an instance (deb-7)
  2. Resize the instance - with a flavour to have more disk space.
  3. After the instance is resized, the instance is permanently set in "ERROR" state - eventhough you can take a console of it and login as usual.

  amande@ZZZZZZZ:~/VMs/IMAGES$ nova list
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+
  | ID                                   | Name                  | Status | Task State | Power State | Networks                  |
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+
  | cc867684-0fe9-48a7-95e9-60890d6e4fd0 | vapp                  | ERROR  | None       | Running     | 172_22-public=172.22.0.49 |
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+

  4. Reset state of the instance.

  amande@ZZZZZZZ:~/VMs/IMAGES$ nova reset-state --active cc867684-0fe9-48a7-95e9-60890d6e4fd0
  amande@axcient:~/VMs/IMAGES$ nova list
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+
  | ID                                   | Name                  | Status | Task State | Power State | Networks                  |
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+
  | cc867684-0fe9-48a7-95e9-60890d6e4fd0 | vapp                  | ACTIVE | None       | Running     | 172_22-public=172.22.0.49 |
  +--------------------------------------+-----------------------+--------+------------+-------------+---------------------------+

  5. After it has its state changed, Soft Reboot the instance.

  6. right at this stage, the drives are not to be seen. and the
  instance will be hung at the boot screen forever- for it cannot find
  any drives to mount.

  LOGS
  --------
  2014-04-24 23:14:48 DEBUG nova.virt.disk.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Checking if we can resize image /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk. size=563714457600 can_resize_image /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:157
  2014-04-24 23:14:48 DEBUG nova.openstack.common.processutils [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Running cmd (subprocess): env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk execute /usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py:147
  2014-04-24 23:14:48 DEBUG nova.openstack.common.processutils [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Running cmd (subprocess): qemu-img resize /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk 563714457600 execute /usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py:147
  2014-04-24 23:14:48 DEBUG nova.virt.disk.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Checking if we can resize filesystem inside /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk. CoW=True is_image_partitionless /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:171
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Instance for image imgfile=/var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk imgfmt=qcow2 partition=None instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:31
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Trying to import guestfs instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:34
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Using primary VFSGuestFS instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:41
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.guestfs [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Setting up appliance for /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk qcow2 setup /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/guestfs.py:111
  2014-04-24 23:14:48 DEBUG nova.virt.disk.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Unable to mount image /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk with error Error mounting /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
  errno: File exists

  If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
  environment variable.  There may also be errors printed above.). Cannot resize. is_image_partitionless /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:183
  2014-04-24 23:14:48 DEBUG nova.openstack.common.lockutils [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Released file lock "/var/lib/nova/instances/_base/6307ef1831bec2fcbd92538a993d622b45ff582a" at /var/lib/nova/instances/locks/nova-_var_lib_nova_instances__base_6307ef1831bec2fcbd92538a993d622b45ff582a lock /usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py:208
  2014-04-24 23:14:48 INFO nova.virt.libvirt.driver [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c] [instance: cc867684-0fe9-48a7-95e9-60890d6e4fd0]  Injecting key into image 04c45b4b-707a-4b12-b4e2-cf5cca48a7b9
  2014-04-24 23:14:48 DEBUG nova.virt.disk.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Inject data image=/var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk key=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmufIqo/G3IdTfErcfmsl1yrBROC1UTWe81vp/1Iqs07tMc7UJWvY1UXNwMXI21QvK6ZMqQyRDdJbN1+U97Xag9hAtePoJaxAwKJZb1UZVvpB23mCLHgqWwVFN7pYKfqv8EXU5FnvSFIXIIibWCUDMbKU9PKR2jzaQ86hDttkNCEIkIXvjgkJAZ/+ySLVzAjgMejwPg3yK8bX7LV+P1BOjFywXJ0aaIy3qk58Zis3v04pIBqwoMo6PZQWNdv+gZmTzaLypqHA/4V1Sls7N8jxaTiJKTuyV0pBu5BtA13dsqGVU6cZONHiw4VnbvICCJFTPzz+iZbj0XCtnwLio6G0B amandes@axcient-amande net=None metadata=[] admin_password=<SANITIZED> files=[] partition=-1 use_cow=True inject_data /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:317
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Instance for image imgfile=/var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk imgfmt=qcow2 partition=-1 instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:31
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Trying to import guestfs instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:34
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Using primary VFSGuestFS instance_for_image /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/api.py:41
  2014-04-24 23:14:48 DEBUG nova.virt.disk.vfs.guestfs [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Setting up appliance for /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk qcow2 setup /usr/lib/python2.6/site-packages/nova/virt/disk/vfs/guestfs.py:111
  2014-04-24 23:14:48 WARNING nova.virt.disk.api [req-99f896e7-549e-4d70-9a84-ee54cf54eaf3 Amrita Mande df9536c185814408adce9b8c1cafcf1c]  Ignoring error injecting data into image (Error mounting /var/lib/nova/instances/cc867684-0fe9-48a7-95e9-60890d6e4fd0/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
  errno: File exists

  If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
  environment variable.  There may also be errors printed above.))

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1312874/+subscriptions
References

[Bug 1312874] [NEW] resizing an instance - causes the drives to dissapear - With it hung during reboot
From: Amrita Mande, 2014-04-25