yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1688038] [NEW] test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 03 May 2017 16:28:23 -0000
Reply-to: Bug 1688038 <1688038@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
full-ubuntu-xenial/a277636/console.html#_2017-05-02_18_16_55_064917

http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/a277636/logs/screen-n-cpu.txt.gz#_May_02_17_49_07_771248

May 02 17:49:07.771248 ubuntu-xenial-rax-ord-8683720 nova-compute[23706]: ERROR oslo_messaging.rpc.server [req-f281fc56-b69f-4e1a-a6d0-752871138ace tempest-ServerRescueTestJSON-709146246 tempest-ServerRescueTestJSON-709146246] Exception during message handling
                                                                          ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                          ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                          ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                          ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                          ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                          ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                          ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                          ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 187, in decorated_function
                                                                          ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                          ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                          ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 156, in decorated_function
                                                                          ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                          ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                          ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                          ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                          ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                          ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                          ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                          ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 3406, in rescue_instance
                                                                          ERROR oslo_messaging.rpc.server     reason=_("Driver Error: %s") % e)
                                                                          ERROR oslo_messaging.rpc.server InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process
                                                                          ERROR oslo_messaging.rpc.server

This is the libvirt guest qemu log:

http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
full-ubuntu-xenial/a277636/logs/libvirt/qemu/instance-00000055.txt.gz

This is where the guest failed to connect to the monitor in the libvirtd
logs:

http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/a277636/logs/libvirt/libvirtd.txt.gz#_2017-05-02_17_49_05_050

Shortly before that:

http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/a277636/logs/libvirt/libvirtd.txt.gz#_2017-05-02_17_49_05_047

2017-05-02 17:49:05.047+0000: 1193: debug : processMonitorEOFEvent:4541
: Monitor connection to 'instance-00000055' closed without SHUTDOWN
event; assuming the domain crashed

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1688038

Title:
  test_rescued_vm_add_remove_security_group fails with
  "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b
  cannot be rescued: Driver Error: failed to connect to monitor socket:
  No such process"

Status in OpenStack Compute (nova):
  New

Bug description:
  http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
  full-ubuntu-xenial/a277636/console.html#_2017-05-02_18_16_55_064917

  http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/a277636/logs/screen-n-cpu.txt.gz#_May_02_17_49_07_771248

  May 02 17:49:07.771248 ubuntu-xenial-rax-ord-8683720 nova-compute[23706]: ERROR oslo_messaging.rpc.server [req-f281fc56-b69f-4e1a-a6d0-752871138ace tempest-ServerRescueTestJSON-709146246 tempest-ServerRescueTestJSON-709146246] Exception during message handling
                                                                            ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                            ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                            ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                            ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                            ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                            ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                            ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                            ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 187, in decorated_function
                                                                            ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                            ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                            ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 156, in decorated_function
                                                                            ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                            ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                            ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                            ERROR oslo_messaging.rpc.server     self.force_reraise()
                                                                            ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                            ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                            ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
                                                                            ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 3406, in rescue_instance
                                                                            ERROR oslo_messaging.rpc.server     reason=_("Driver Error: %s") % e)
                                                                            ERROR oslo_messaging.rpc.server InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process
                                                                            ERROR oslo_messaging.rpc.server

  This is the libvirt guest qemu log:

  http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
  full-ubuntu-xenial/a277636/logs/libvirt/qemu/instance-00000055.txt.gz

  This is where the guest failed to connect to the monitor in the
  libvirtd logs:

  http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/a277636/logs/libvirt/libvirtd.txt.gz#_2017-05-02_17_49_05_050

  Shortly before that:

  http://logs.openstack.org/73/461473/3/check/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/a277636/logs/libvirt/libvirtd.txt.gz#_2017-05-02_17_49_05_047

  2017-05-02 17:49:05.047+0000: 1193: debug :
  processMonitorEOFEvent:4541 : Monitor connection to
  'instance-00000055' closed without SHUTDOWN event; assuming the domain
  crashed

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1688038/+subscriptions
Follow ups

[Bug 1688038] Re: test_rescued_vm_add_remove_security_group fails with "InstanceNotRescuable: Instance 4869b462-c3cf-4437-8c94-1d0dcd5fff8b cannot be rescued: Driver Error: failed to connect to monitor socket: No such process"
From: Launchpad Bug Tracker, 2017-09-27