yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #18252
[Bug 1349978] [NEW] hard reboot doesn't re-create instance folder
Public bug reported:
hard-rebooting a libvirt instance assumes the instance's folder exists
(ie. /var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/).
When nova tries to re-create the XML file it causes this stack trace:
2014-07-29 16:02:41.250 2922 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2649, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher self._set_instance_obj_error_state(context, instance)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2630, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher bad_volumes_callback=bad_volumes_callback)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1999, in reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2092, in _hard_reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher write_to_disk=True)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3455, in to_xml
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher libvirt_utils.write_to_file(xml_path, xml)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 526, in write_to_file
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher with open(path, 'w') as f:
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
If I manually create the directory hard-reboot works as expected.
Example why this might happen:
If your nova-compute node was stateless (booted with ram_fs mounted on /) it would forget all its instances after being power cycled. The instances could be recovered with hard-reboot (assuming their disks weren't local) if the instance's folder was re-created.
nova-compute version: icehouse 2014.1.1
I could write a patch + tests if we decide to do this.
** Affects: nova
Importance: Undecided
Status: New
** Tags: compute
** Description changed:
hard-rebooting a libvirt instance assumes the instance's folder exists
(ie. /var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/).
When nova tries to re-create the XML file it causes this stack trace:
2014-07-29 16:02:41.250 2922 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2649, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher self._set_instance_obj_error_state(context, instance)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2630, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher bad_volumes_callback=bad_volumes_callback)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1999, in reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2092, in _hard_reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher write_to_disk=True)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3455, in to_xml
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher libvirt_utils.write_to_file(xml_path, xml)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 526, in write_to_file
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher with open(path, 'w') as f:
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
If I manually create the directory hard-reboot works as expected.
Example why this might happen:
If your nova-compute node was stateless (booted with ram_fs mounted on /) it would forget all its instances after being power cycled. The instances could be recovered with hard-reboot (assuming their disks weren't local) if the instance's folder was re-created.
nova-compute version: icehouse 2014.1.1
-
- I could write a patch + tests if we decide to do this. The change would be small.
+ I could write a patch + tests if we decide to do this.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1349978
Title:
hard reboot doesn't re-create instance folder
Status in OpenStack Compute (Nova):
New
Bug description:
hard-rebooting a libvirt instance assumes the instance's folder exists
(ie. /var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/).
When nova tries to re-create the XML file it causes this stack trace:
2014-07-29 16:02:41.250 2922 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2649, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher self._set_instance_obj_error_state(context, instance)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2630, in reboot_instance
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher bad_volumes_callback=bad_volumes_callback)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1999, in reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2092, in _hard_reboot
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher write_to_disk=True)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3455, in to_xml
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher libvirt_utils.write_to_file(xml_path, xml)
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 526, in write_to_file
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher with open(path, 'w') as f:
2014-07-29 16:02:41.250 2922 TRACE oslo.messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/var/lib/nova/instances/e4e4a4a3-1036-4734-9c0b-3bb34c88b8b6/libvirt.xml'
If I manually create the directory hard-reboot works as expected.
Example why this might happen:
If your nova-compute node was stateless (booted with ram_fs mounted on /) it would forget all its instances after being power cycled. The instances could be recovered with hard-reboot (assuming their disks weren't local) if the instance's folder was re-created.
nova-compute version: icehouse 2014.1.1
I could write a patch + tests if we decide to do this.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1349978/+subscriptions
Follow ups
References