yahoo-eng-team team mailing list archive

[Ankit Agrawal <ankit11.agrawal@xxxxxxxxxxx>]

Public bug reported:

Single Node Environment:
When we resize an instance in single node environment, instance directory is renamed to “<instance_id>_resize” directory.
Then a new directory is created with the instance path on the same node and all instance files with new configuration will be copied to this directory.
If there is not enough space left on this node to create or copying instance files then disk full error occurs and VM goes into "error" state.

Multi Node Environment:
When we resize an instance in multi node environment, directory is renamed to “<instance_id>_resize” directory on source node. 
Then a new directory is created with the instance path on the destination node and all instance files with new configuration will be copied to this directory.
If there is not enough space left on destination node to create or copying instance files then disk full error occurs and VM goes to "error" state.

As of now in both the cases there is no way to recover instance from the error state which is not good from end users perspective.
Ideally it should log the warning if disk space is full and should keep the instance up and running with old configuration.

Stack trace for reference:
2014-07-22 03:50:04.618 ERROR nova.compute.manager [req-d45a250c-bef9-4b42-96d3-187296dba8b0 admin demo] [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Setting instance vm_state to ERROR
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Traceback (most recent call last):
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3656, in finish_resize
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     disk_info, image)
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3624, in _finish_resize
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info, power_on)
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5053, in finish_migration
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info=None, inject_files=False)
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2559, in _create_image
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/openstack/common/fileutils.py", line 35, in ensure_tree
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     os.makedirs(path)
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     mkdir(name, mode)
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] OSError: [Errno 28] No space left on device: '/mnt/stack/data/nova/instances/30003b95-d88a-41f7-8ee2-44bffc524576'
2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]

** Affects: nova
     Importance: Undecided
     Assignee: Ankit Agrawal (ankitagrawal)
         Status: New


** Tags: ntt

** Changed in: nova
     Assignee: (unassigned) => Ankit Agrawal (ankitagrawal)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1353343

Title:
  Cannot recover instance if error occurs during resize

Status in OpenStack Compute (Nova):
  New

Bug description:
  Single Node Environment:
  When we resize an instance in single node environment, instance directory is renamed to “<instance_id>_resize” directory.
  Then a new directory is created with the instance path on the same node and all instance files with new configuration will be copied to this directory.
  If there is not enough space left on this node to create or copying instance files then disk full error occurs and VM goes into "error" state.

  Multi Node Environment:
  When we resize an instance in multi node environment, directory is renamed to “<instance_id>_resize” directory on source node. 
  Then a new directory is created with the instance path on the destination node and all instance files with new configuration will be copied to this directory.
  If there is not enough space left on destination node to create or copying instance files then disk full error occurs and VM goes to "error" state.

  As of now in both the cases there is no way to recover instance from the error state which is not good from end users perspective.
  Ideally it should log the warning if disk space is full and should keep the instance up and running with old configuration.

  Stack trace for reference:
  2014-07-22 03:50:04.618 ERROR nova.compute.manager [req-d45a250c-bef9-4b42-96d3-187296dba8b0 admin demo] [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Setting instance vm_state to ERROR
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] Traceback (most recent call last):
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3656, in finish_resize
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     disk_info, image)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/compute/manager.py", line 3624, in _finish_resize
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info, power_on)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5053, in finish_migration
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     block_device_info=None, inject_files=False)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2559, in _create_image
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/opt/stack/nova/nova/openstack/common/fileutils.py", line 35, in ensure_tree
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     os.makedirs(path)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]     mkdir(name, mode)
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576] OSError: [Errno 28] No space left on device: '/mnt/stack/data/nova/instances/30003b95-d88a-41f7-8ee2-44bffc524576'
  2014-07-22 03:50:04.618 TRACE nova.compute.manager [instance: 30003b95-d88a-41f7-8ee2-44bffc524576]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1353343/+subscriptions