yahoo-eng-team team mailing list archive

[HYSong <1919401@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Env info:
openstack version: rocky
storage back-end: ceph
hypervisor: qemu/KVM

Sample traceback:
==================================
[req-69c94c9a-6ee4-4936-8ce5-9a23b7aea89a 00b865b2a29e47f8b57a62ac624bdfa4 9edd1f98bf2f47e885f7077a066c83dd - default default] 
[instance: 642ab2df-4dc2-4ca8-9bbd-ab19c72352df] 
Setting instance vm_state to ERROR: OSError: [Errno 39] Directory not empty
Traceback (most recent call last):
  File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
    yield
  File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4693, in _resize_instance
    timeout, retry_interval)
  File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8668, in migrate_disk_and_power_off
    shared_storage)
  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8622, in migrate_disk_and_power_off
    os.rename(inst_base, inst_base_resize)
OSError: [Errno 39] Directory not empty


Description:
==================================
1. Executing VM resize error, and the the dir of inst_base_resize has been created by `os.rename(inst_base, inst_base_resize)` in the function of migrate_disk_and_power_off. The function of `_error_out_instance_on_exception` in _resize_instance just catch exceptions and can not rollback dir.

2. Executing command of `openstack server set` to recover VM status to
active.

3. Executing VM resize error again, and Exception in Sample traceback.
The operation of `os.rename(inst_base, inst_base_resize)` failed because
of the dir of inst_base_resize has console.log.

4. Whether or not execute `self._cleanup_remote_migration` before
`os.rename(inst_base, inst_base_resize)`?  Is there any methods to
optimize exception handling in the function of _resize_instance ?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1919401

Title:
  the function of _resize_instance lack of exception handling

Status in OpenStack Compute (nova):
  New

Bug description:
  Env info:
  openstack version: rocky
  storage back-end: ceph
  hypervisor: qemu/KVM

  Sample traceback:
  ==================================
  [req-69c94c9a-6ee4-4936-8ce5-9a23b7aea89a 00b865b2a29e47f8b57a62ac624bdfa4 9edd1f98bf2f47e885f7077a066c83dd - default default] 
  [instance: 642ab2df-4dc2-4ca8-9bbd-ab19c72352df] 
  Setting instance vm_state to ERROR: OSError: [Errno 39] Directory not empty
  Traceback (most recent call last):
    File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 8333, in _error_out_instance_on_exception
      yield
    File "/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py", line 4693, in _resize_instance
      timeout, retry_interval)
    File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8668, in migrate_disk_and_power_off
      shared_storage)
    File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/var/lib/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8622, in migrate_disk_and_power_off
      os.rename(inst_base, inst_base_resize)
  OSError: [Errno 39] Directory not empty

  
  Description:
  ==================================
  1. Executing VM resize error, and the the dir of inst_base_resize has been created by `os.rename(inst_base, inst_base_resize)` in the function of migrate_disk_and_power_off. The function of `_error_out_instance_on_exception` in _resize_instance just catch exceptions and can not rollback dir.

  2. Executing command of `openstack server set` to recover VM status to
  active.

  3. Executing VM resize error again, and Exception in Sample traceback.
  The operation of `os.rename(inst_base, inst_base_resize)` failed
  because of the dir of inst_base_resize has console.log.

  4. Whether or not execute `self._cleanup_remote_migration` before
  `os.rename(inst_base, inst_base_resize)`?  Is there any methods to
  optimize exception handling in the function of _resize_instance ?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1919401/+subscriptions