← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #57577

[Bug 1631481] [NEW] Revert resize does not delete instance directory with Ceph

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Resize revertion leaves instance directory on the second host with Ceph
image backend. As the result the second attempt to resize the instance
to the same host fails with n-cpu.log:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize
    disk_info, image_meta)
  File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize
    old_instance_type)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize
    block_device_info, power_on)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration
    self._ensure_console_log_for_instance(instance)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance
    libvirt_utils.file_open(console_file, 'a').close()
  File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open
    return open(*args, **kwargs)
IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log'

$ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/
total 24
-rw-rw-r--. 1 qemu  qemu     19342 Oct  7 21:23 console.log
-rw-rw-r--. 1 stack libvirtd  2762 Oct  7 21:22 libvirt.xml

Steps to reproduce:
1 Run 2-nodes devstack with Ceph image backend
2 Run an instance
 $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
3 Disable the instance host
 $ nova service-disable 172.16.1.10 nova-compute
4 Resize the instance to another host
 $ nova migrate inst-1
5 Revert resize
 $ nova resize-revert inst-1
6 Resize the instance again
 $ nova migrate inst-1
7 Check the instance state

Actual result - the instance is in error state.
Expected result - the instance is in verify_resize state.

Check n-cpu.log on the second node, where the instance was migrated.

This has been reproduced on master
commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015
Merge: 870a77f 453e71d
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Wed Oct 5 01:35:48 2016 +0000

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Resize revertion leaves instance directory on the second host with Ceph
  image backend. As the result the second attempt to resize the instance
  to the same host fails with n-cpu.log:
  
  Traceback (most recent call last):
-   File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize
-     disk_info, image_meta)
-   File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize
-     old_instance_type)
-   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
-     self.force_reraise()
-   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
-     six.reraise(self.type_, self.value, self.tb)
-   File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize
-     block_device_info, power_on)
-   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration
-     self._ensure_console_log_for_instance(instance)
-   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance
-     libvirt_utils.file_open(console_file, 'a').close()
-   File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open
-     return open(*args, **kwargs)
+   File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize
+     disk_info, image_meta)
+   File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize
+     old_instance_type)
+   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
+     self.force_reraise()
+   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
+     six.reraise(self.type_, self.value, self.tb)
+   File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize
+     block_device_info, power_on)
+   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration
+     self._ensure_console_log_for_instance(instance)
+   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance
+     libvirt_utils.file_open(console_file, 'a').close()
+   File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open
+     return open(*args, **kwargs)
  IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log'
  
  $ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/
  total 24
  -rw-rw-r--. 1 qemu  qemu     19342 Oct  7 21:23 console.log
  -rw-rw-r--. 1 stack libvirtd  2762 Oct  7 21:22 libvirt.xml
  
  Steps to reproduce:
  1 Run 2-nodes devstack with Ceph image backend
  2 Run an instance
-  $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
+  $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
  3 Disable the instance host
-  $ nova service-disable 172.16.1.10 nova-compute
+  $ nova service-disable 172.16.1.10 nova-compute
  4 Resize the instance to another host
-  $ nova migrate inst-1
+  $ nova migrate inst-1
  5 Revert resize
-  $ nova resize-revert inst-1
+  $ nova resize-revert inst-1
  6 Resize the instance again
-  $ nova migrate inst-1
+  $ nova migrate inst-1
  7 Check the instance state
  
  Actual result - the instance is in error state.
  Expected result - the instance is in verify_resize state.
  
  Check n-cpu.log on the second node, where the instance was migrated.
+ 
+ This has been reproduced on master
+ commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015
+ Merge: 870a77f 453e71d
+ Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
+ Date:   Wed Oct 5 01:35:48 2016 +0000

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1631481

Title:
  Revert resize does not delete instance directory with Ceph

Status in OpenStack Compute (nova):
  New

Bug description:
  Resize revertion leaves instance directory on the second host with
  Ceph image backend. As the result the second attempt to resize the
  instance to the same host fails with n-cpu.log:

  Traceback (most recent call last):
    File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize
      disk_info, image_meta)
    File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize
      old_instance_type)
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize
      block_device_info, power_on)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration
      self._ensure_console_log_for_instance(instance)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance
      libvirt_utils.file_open(console_file, 'a').close()
    File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open
      return open(*args, **kwargs)
  IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log'

  $ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/
  total 24
  -rw-rw-r--. 1 qemu  qemu     19342 Oct  7 21:23 console.log
  -rw-rw-r--. 1 stack libvirtd  2762 Oct  7 21:22 libvirt.xml

  Steps to reproduce:
  1 Run 2-nodes devstack with Ceph image backend
  2 Run an instance
   $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
  3 Disable the instance host
   $ nova service-disable 172.16.1.10 nova-compute
  4 Resize the instance to another host
   $ nova migrate inst-1
  5 Revert resize
   $ nova resize-revert inst-1
  6 Resize the instance again
   $ nova migrate inst-1
  7 Check the instance state

  Actual result - the instance is in error state.
  Expected result - the instance is in verify_resize state.

  Check n-cpu.log on the second node, where the instance was migrated.

  This has been reproduced on master
  commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015
  Merge: 870a77f 453e71d
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Wed Oct 5 01:35:48 2016 +0000

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1631481/+subscriptions
  Thread Previous • Date Previous • • Thread Next