yahoo-eng-team team mailing list archive

["Markus Zoeller \(markus_z\)" <mzoeller@xxxxxxxxxxxxxxxxxx>]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY


** Changed in: nova
   Importance: Low => Undecided

** Changed in: nova
       Status: Confirmed => Expired

** Changed in: nova
     Assignee: Ankit Agrawal (ankitagrawal) => (unassigned)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1353343

Title:
  Cannot recover instance if error occurs during resize

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Single Node Environment:
  When we resize an instance in single node environment, instance directory is renamed to “<instance_id>_resize” directory.
  Then a new directory is created with the instance path on the same node and all instance files with new configuration will be copied to this directory.
  If there is not enough space left on this node to create or copying instance files then disk full error occurs and VM goes into "error" state.

  Multi Node Environment:
  When we resize an instance in multi node environment, directory is renamed to “<instance_id>_resize” directory on source node. 
  Then a new directory is created with the instance path on the destination node and all instance files with new configuration will be copied to this directory.
  If there is not enough space left on destination node to create or copying instance files then disk full error occurs and VM goes to "error" state.

  As of now in both the cases there is no way to recover instance from the error state which is not good from end users perspective.
  Ideally it should log the warning if disk space is full and should keep the instance up and running with old configuration.

  Stack trace for reference:
  2014-07-22 03:50:04.618 ERROR nova.compute.manager [req-d45a250c-bef9-4b42-96d3-187296dba8b0 admin demo] [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Setting instance vm_state to ERROR
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Traceback (most recent call last):
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3656, in finish_resize
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     disk_info, image)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3624, in _finish_resize
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info, power_on)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5053, in finish_migration
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info=None, inject_files=False)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2559, in _create_image
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/openstack/common/fileutils.py", line 35, in ensure_tree
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     os.makedirs(path)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     mkdir(name, mode)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] OSError: [Errno 28] No space left on device: '/mnt/stack/data/nova/instances/30003b95-d88a-41f7-8ee2-44bffc524576'
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1353343/+subscriptions