yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1970642] Re: evacuation failure causes POST_FAILURE in the nova-live-migration job

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1970642@xxxxxxxxxxxxxxxxxx>
Date: Tue, 05 Dec 2023 11:38:29 -0000
Reply-to: Bug 1970642 <1970642@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/nova/+/901815
Committed: https://opendev.org/openstack/nova/commit/1f8ce3cc70ceaf8edd25b1ae2646e525cf2b3973
Submitter: "Zuul (22348)"
Branch:    master

commit 1f8ce3cc70ceaf8edd25b1ae2646e525cf2b3973
Author: Rajesh Tailor <ratailor@xxxxxxxxxx>
Date:   Fri Nov 24 13:25:25 2023 +0530

    Add retry on attachment_update
    
    In nova-live-migration job, evacuation failures are causing
    POST_FAILURE.
    
    As per discussion on bug, it looks like 500 is coming from cinder
    which is cause of this failure.
    
    Similar to attachment_delete method, this change adds a retry
    mechanism in cinder API calls attachment_update method.
    
    Closes-Bug: #1970642
    Change-Id: I1da3c8481f7e7a1e8776cf03f5c4cf117b7fabaa


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1970642

Title:
  evacuation failure causes POST_FAILURE in the nova-live-migration job

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  The nova-live-migration job has a post task to test evacuation. This
  test fails intermittently causing the job to fail with POST_FAILURE
  status.

  Evacuation fails due to libvirt connection error.

  job log:

  2022-04-27 14:24:41.092806 | controller | + /opt/stack/nova/roles/run-
  evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 :
  echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE
  status'

  2022-04-27 14:24:41.092962 | controller | Timed out waiting for server
  evacuate-bfv-test to go to ACTIVE status

  nova-compute log:

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server [None
  req-a012194e-8201-43fa-ae56-a9194a5fd5ad demo admin] Exception during
  message handling: nova.exception.HypervisorUnavailable: Connection to
  the hypervisor is broken on host

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
  call last):

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in
  get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     conn =
  self._get_connection()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in
  _get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self._queue_conn_event_handler(

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in
  _get_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     self._wrapped_conn
  = self._get_new_connection()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in
  _get_new_connection

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     wrapped_conn =
  self._connect(self._uri, self._read_only)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._libvirt_proxy.openAuth(uri, auth, flags)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193,
  in doit

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151,
  in proxy_call

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     rv = execute(f,
  *args, **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
  in execute

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     six.reraise(c, e,
  tb)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
  in tworker

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     rv = meth(*args,
  **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if ret is
  None:raise libvirtError('virConnectOpenAuth() failed')

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server libvirt.libvirtError:
  Failed to connect socket to '/var/run/libvirt/libvirt-sock':
  Connection refused

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server During handling of the
  above exception, another exception occurred:

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
  call last):

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 165, in _process_incoming

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     res =
  self.dispatcher.dispatch(message)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._do_dispatch(endpoint, method, ctxt, args)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     result =
  func(ctxt, **new_args)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
  line 241, in inner

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return func(*args,
  **kwargs)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  _emit_versioned_exception_notification(

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return f(self,
  context, *args, **kw)

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 184, in
  decorated_function

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  LOG.warning("Failed to revert task state for instance. "

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 155, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/utils.py", line 1439, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 212, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  compute_utils.add_instance_fault_from_exc(context,

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  227, in __exit__

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self.force_reraise()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
  200, in force_reraise

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise self.value

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 201, in
  decorated_function

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  function(self, context, *args, **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3548, in
  rebuild_instance

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  self._do_rebuild_instance_with_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/manager.py", line 3625, in
  _do_rebuild_instance_with_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     claim_context =
  rebuild_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/usr/local/lib/python3.8/dist-
  packages/oslo_concurrency/lockutils.py", line 391, in inner

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return f(*args,
  **kwargs)

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 204, in
  rebuild_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  self._move_claim(

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 280, in
  _move_claim

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if
  self.disabled(nodename):

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/compute/resource_tracker.py", line 666, in
  disabled

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     not
  self.driver.node_is_available(nodename))

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/driver.py", line 1564, in node_is_available

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     if nodename in
  self.get_available_nodes():

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 11037, in
  get_available_nodes

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     return
  [self._host.get_hostname()]

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 1051, in
  get_hostname

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     hostname =
  self.get_connection().getHostname()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server   File
  "/opt/stack/nova/nova/virt/libvirt/host.py", line 596, in
  get_connection

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server     raise
  exception.HypervisorUnavailable()

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server
  nova.exception.HypervisorUnavailable: Connection to the hypervisor is
  broken on host

  Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
  compute[97475]: ERROR oslo_messaging.rpc.server

  Example failure:
  https://zuul.opendev.org/t/openstack/build/6c792e4ee72542cbaf386710e57a4b01/log/controller/logs/screen-
  n-cpu.txt#12329

  I see multiple matching failures from the last two weeks:
  https://paste.opendev.org/show/bbB6cuBQBGSUPyAxPXva/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1970642/+subscriptions
References

[Bug 1970642] [NEW] evacuation failure causes POST_FAILURE in the nova-live-migration job
From: Balazs Gibizer, 2022-04-27