kernel-packages team mailing list archive

Thread
Date
[Bug 1536280] Re: domain shutdown fails for libvirt/lxc

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Thomas Maddox <thomas.maddox@xxxxxxxxxxxxx>
Date: Wed, 20 Jan 2016 18:57:33 -0000
Reply-to: Bug 1536280 <1536280@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Description changed:

  There appears to be a regression from kernel 3.13.0-58-generic where
  virDomainShutdown no longer brings the domain to shutoff state in newer
  Ubuntu kernels.
  
  Details:
+ ========
  
  OS:
+ ---
  $ lsb_release -rd
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04
  
  Kernel:
+ -------
  $ uname -a
  Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  
  Package:
+ --------
  $ dpkg -l | grep libvirt-bin
  ii  libvirt-bin                          1.2.2-0ubuntu13.1.16                  amd64        programs for the libvirt library
  
- 
- Kernel: 3.13.0-76-generic
- Libvirt Version: 1.2.2
  Libvirt virt driver: LXC
  
- 
  Issue seen from console:
+ ------------------------
  
  $ sudo virsh -c lxc:/// shutdown instance-00000001
  error: Failed to shutdown domain instance-00000001
  error: internal error: Child process (1304) unexpected exit status 1
  
- 
  libvirtd logs:
+ --------------
  
  2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
  2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
  
- 
- nova-compute logs (when attempting to do a nova stop <instance> in devstack):
+ nova-compute logs
+ ----------------- 
+ *Note*: From when attempting to do a nova stop <instance> in devstack
  
  2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
  (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     do_stop_instance()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._power_off_instance(context, instance, clean_shutdown)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self.driver.power_off(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._clean_shutdown(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     guest.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._domain.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
  
  This all worked gracefully on 3.13.0-58-generic, as well as the newer
  3.13.x kernels, including upstream 3.13.0 installed from here:
  http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-
  image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb.

** Description changed:

  There appears to be a regression from kernel 3.13.0-58-generic where
  virDomainShutdown no longer brings the domain to shutoff state in newer
  Ubuntu kernels.
  
  Details:
  ========
  
  OS:
  ---
  $ lsb_release -rd
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04
  
  Kernel:
  -------
  $ uname -a
  Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  
  Package:
  --------
  $ dpkg -l | grep libvirt-bin
  ii  libvirt-bin                          1.2.2-0ubuntu13.1.16                  amd64        programs for the libvirt library
  
  Libvirt virt driver: LXC
  
  Issue seen from console:
  ------------------------
  
  $ sudo virsh -c lxc:/// shutdown instance-00000001
  error: Failed to shutdown domain instance-00000001
  error: internal error: Child process (1304) unexpected exit status 1
  
  libvirtd logs:
  --------------
  
  2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
  2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1
  
  nova-compute logs
- ----------------- 
+ -----------------
  *Note*: From when attempting to do a nova stop <instance> in devstack
  
+ ```
  2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
  (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     do_stop_instance()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._power_off_instance(context, instance, clean_shutdown)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self.driver.power_off(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._clean_shutdown(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     guest.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._domain.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher
+ ```
  
  This all worked gracefully on 3.13.0-58-generic, as well as the newer
  3.13.x kernels, including upstream 3.13.0 installed from here:
  http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-
  image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1536280

Title:
  domain shutdown fails for libvirt/lxc

Status in libvirt package in Ubuntu:
  New
Status in linux package in Ubuntu:
  Incomplete

Bug description:
  There appears to be a regression from kernel 3.13.0-58-generic where
  virDomainShutdown no longer brings the domain to shutoff state in
  newer Ubuntu kernels.

  Details:
  ========

  OS:
  ---
  $ lsb_release -rd
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04

  Kernel:
  -------
  $ uname -a
  Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  Package:
  --------
  $ dpkg -l | grep libvirt-bin
  ii  libvirt-bin                          1.2.2-0ubuntu13.1.16                  amd64        programs for the libvirt library

  Libvirt virt driver: LXC

  Issue seen from console:
  ------------------------

  $ sudo virsh -c lxc:/// shutdown instance-00000001
  error: Failed to shutdown domain instance-00000001
  error: internal error: Child process (1304) unexpected exit status 1

  libvirtd logs:
  --------------

  2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
  2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1

  nova-compute logs
  -----------------
  *Note*: From when attempting to do a nova stop <instance> in devstack

  
  2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
  (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     do_stop_instance()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._power_off_instance(context, instance, clean_shutdown)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self.driver.power_off(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._clean_shutdown(instance, timeout, retry_interval)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     guest.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     self._domain.shutdown()
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
  2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher

  
  This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1536280/+subscriptions