yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1349978] Re: hard reboot doesn't re-create instance folder

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 18 Dec 2014 20:01:26 -0000
Reply-to: Bug 1349978 <1349978@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1349978

Title:
  hard reboot doesn't re-create instance folder

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I'm running stateless Nova compute servers (booted with ram_fs mounted
  on /). If I power-cycle a server it forgets all its instances but they
  could be recovered by hard-rebooting, except hard-rebooting a libvirt
  instance assumes the instance's folder already exists (ie.
  /var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/). When
  nova tries to re-create the XML file it causes this stack trace:

  2014-07-29 16:02:41.250 2922 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     pass
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     function(self, context, *args, **kwargs)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2649, in reboot_instance
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     self._set_instance_obj_error_state(context, instance)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2630, in reboot_instance
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     bad_volumes_callback=bad_volumes_callback)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1999, in reboot
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     block_device_info)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2092, in _hard_reboot
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     write_to_disk=True)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3455, in to_xml
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     libvirt_utils.write_to_file(xml_path, xml)
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 526, in write_to_file
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher     with open(path, 'w') as f:
  2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'

  If I manually create the directory then hard-reboot will re-create
  everything else and work as expected.

  OS: Ubuntu 14.04, 64-bit
  nova-compute version: icehouse 2014.1.1
  libvirt: v1.2.2
  Using Ceph for storage: v0.80.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1349978/+subscriptions
References

[Bug 1349978] [NEW] hard reboot doesn't re-create instance folder
From: HT_Sergio, 2014-07-29