yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1970642] [NEW] evacuation failure causes POST_FAILURE in the nova-live-migration job

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1970642@xxxxxxxxxxxxxxxxxx>
Date: Wed, 27 Apr 2022 16:59:43 -0000
Reply-to: Bug 1970642 <1970642@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The nova-live-migration job has a post task to test evacuation. This
test fails intermittently causing the job to fail with POST_FAILURE
status.

Evacuation fails due to libvirt connection error.

job log:

2022-04-27 14:24:41.092806 | controller | + /opt/stack/nova/roles/run-
evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 :
echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE
status'

2022-04-27 14:24:41.092962 | controller | Timed out waiting for server
evacuate-bfv-test to go to ACTIVE status

nova-compute log:

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server [None
req-a012194e-8201-43fa-ae56-a9194a5fd5ad demo admin] Exception during
message handling: nova.exception.HypervisorUnavailable: Connection to
the hypervisor is broken on host

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in get_connection

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     conn =
self._get_connection()

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in
_get_connection

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self._queue_conn_event_handler(

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     self.force_reraise()

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in
_get_connection

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     self._wrapped_conn =
self._get_new_connection()

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in
_get_new_connection

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     wrapped_conn =
self._connect(self._uri, self._read_only)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
self._libvirt_proxy.openAuth(uri, auth, flags)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193, in
doit

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     result =
proxy_call(self._autowrap, f, *args, **kwargs)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151, in
proxy_call

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     rv = execute(f,
*args, **kwargs)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in
execute

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
tb)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise value

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in
tworker

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
**kwargs)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     if ret is None:raise
libvirtError('virConnectOpenAuth() failed')

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server libvirt.libvirtError:
Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection
refused

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server During handling of the
above exception, another exception occurred:

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     res =
self.dispatcher.dispatch(message)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
self._do_dispatch(endpoint, method, ctxt, args)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     result = func(ctxt,
**new_args)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 241, in inner

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return func(*args,
**kwargs)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     self.force_reraise()

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return f(self,
context, *args, **kw)

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 184, in
decorated_function

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     LOG.warning("Failed
to revert task state for instance. "

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     self.force_reraise()

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 155, in
decorated_function

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/utils.py", line 1439, in
decorated_function

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 212, in
decorated_function

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     self.force_reraise()

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 201, in
decorated_function

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
function(self, context, *args, **kwargs)

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 3548, in
rebuild_instance

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self._do_rebuild_instance_with_claim(

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/manager.py", line 3625, in
_do_rebuild_instance_with_claim

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     claim_context =
rebuild_claim(

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py",
line 391, in inner

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return f(*args,
**kwargs)

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 204, in
rebuild_claim

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
self._move_claim(

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 280, in
_move_claim

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     if
self.disabled(nodename):

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 666, in
disabled

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     not
self.driver.node_is_available(nodename))

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/driver.py", line 1564, in node_is_available

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     if nodename in
self.get_available_nodes():

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 11037, in
get_available_nodes

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     return
[self._host.get_hostname()]

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 1051, in get_hostname

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     hostname =
self.get_connection().getHostname()

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server   File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 596, in get_connection

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server     raise
exception.HypervisorUnavailable()

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
nova.exception.HypervisorUnavailable: Connection to the hypervisor is
broken on host

Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server

Example failure:
https://zuul.opendev.org/t/openstack/build/6c792e4ee72542cbaf386710e57a4b01/log/controller/logs/screen-
n-cpu.txt#12329

I see multiple matching failures from the last two weeks:
https://paste.opendev.org/show/bbB6cuBQBGSUPyAxPXva/

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1970642

Title:
  evacuation failure causes POST_FAILURE in the nova-live-migration job

Status in OpenStack Compute (nova):
  New

Bug description:
  The nova-live-migration job has a post task to test evacuation. This
  test fails intermittently causing the job to fail with POST_FAILURE
  status.

  Evacuation fails due to libvirt connection error.

  job log:

  2022-04-27 14:24:41.092806 | controller | + /opt/stack/nova/roles/run-
  evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 :
  echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE
  status'

  2022-04-27 14:24:41.092962 | controller | Timed out waiting for server
  evacuate-bfv-test to go to ACTIVE status

  nova-compute log:

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server [None
  req-a012194e-8201-43fa-ae56-a9194a5fd5ad demo admin] Exception during
  message handling: nova.exception.HypervisorUnavailable: Connection to
  the hypervisor is broken on host

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
  call last):

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in
  get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     conn =
  self._get_connection()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in
  _get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self._queue_conn_event_handler(

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in
  _get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     self._wrapped_conn
  = self._get_new_connection()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in
  _get_new_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     wrapped_conn =
  self._connect(self._uri, self._read_only)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._libvirt_proxy.openAuth(uri, auth, flags)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193,
  in doit

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151,
  in proxy_call

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     rv = execute(f,
  *args, **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
  in execute

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
  tb)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
  in tworker

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
  **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if ret is
  None:raise libvirtError('virConnectOpenAuth() failed')

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server libvirt.libvirtError:
  Failed to connect socket to '/var/run/libvirt/libvirt-sock':
  Connection refused

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server During handling of the
  above exception, another exception occurred:

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
  call last):

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 165, in _process_incoming

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     res =
  self.dispatcher.dispatch(message)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._do_dispatch(endpoint, method, ctxt, args)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     result =
  func(ctxt, **new_args)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 241, in inner

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return func(*args,
  **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  _emit_versioned_exception_notification(

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return f(self,
  context, *args, **kw)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 184, in
  decorated_function

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  LOG.warning("Failed to revert task state for instance. "

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 155, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/utils.py", line 1439, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 212, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  compute_utils.add_instance_fault_from_exc(context,

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 201, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3548, in
  rebuild_instance

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self._do_rebuild_instance_with_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3625, in
  _do_rebuild_instance_with_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     claim_context =
  rebuild_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_concurrency/lockutils.py", line 391, in inner

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return f(*args,
  **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 204, in
  rebuild_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._move_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 280, in
  _move_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if
  self.disabled(nodename):

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 666, in
  disabled

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     not
  self.driver.node_is_available(nodename))

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/driver.py", line 1564, in node_is_available

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if nodename in
  self.get_available_nodes():

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 11037, in
  get_available_nodes

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  [self._host.get_hostname()]

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 1051, in
  get_hostname

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     hostname =
  self.get_connection().getHostname()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 596, in
  get_connection

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise
  exception.HypervisorUnavailable()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  nova.exception.HypervisorUnavailable: Connection to the hypervisor is
  broken on host

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Example failure:
  https://zuul.opendev.org/t/openstack/build/6c792e4ee72542cbaf386710e57a4b01/log/controller/logs/screen-
  n-cpu.txt#12329

  I see multiple matching failures from the last two weeks:
  https://paste.opendev.org/show/bbB6cuBQBGSUPyAxPXva/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1970642/+subscriptions
Follow ups

[Bug 1970642] Re: evacuation failure causes POST_FAILURE in the nova-live-migration job
From: OpenStack Infra, 2023-12-05