yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1912310] [NEW] libvirt.libvirtError: unable to connect to server at

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1912310@xxxxxxxxxxxxxxxxxx>
Date: Tue, 19 Jan 2021 09:11:54 -0000
Reply-to: Bug 1912310 <1912310@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

nova-live-migration fails due to the following error on the src during a
migration test:

https://zuul.opendev.org/t/openstack/build/76c37d93ba274cd6b90fbe0659255959/log/compute1/logs/screen-n-cpu.txt

Jan 18 12:03:25.145915 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation has aborted
Jan 18 12:03:25.194466 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: INFO nova.compute.manager [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Swapping old allocation on dict_keys(['022fd821-f027-4019-ac4b-cb1e1d2d5423']) held by migration 7b332f03-b32e-4cc8-8e2d-23ed7bee03bd for instance
Jan 18 12:03:25.239903 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Live Migration failure: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused
Jan 18 12:03:25.240119 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: DEBUG nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation thread notification {{(pid=52916) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:9448}}
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: Traceback (most recent call last):
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     timer()
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     cb(*args, **kw)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 175, in _do_send
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     waiter.switch(result)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 221, in main
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     result = function(*args, **kwargs)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/utils.py", line 660, in context_wrapper
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     return func(*args, **kwargs)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9103, in _live_migration_operation
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     LOG.error("Live Migration failure: %s", e, instance=instance)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     self.force_reraise()
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     six.reraise(self.type_, self.value, self.tb)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise value
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9091, in _live_migration_operation
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     guest.migrate(self._live_migration_uri(dest),
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 649, in migrate
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     self._domain.migrateToURI3(
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 190, in doit
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 148, in proxy_call
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     rv = execute(f, *args, **kwargs)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 129, in execute
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     six.reraise(c, e, tb)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise value
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 83, in tworker
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     rv = meth(*args, **kwargs)
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 2099, in migrateToURI3
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise libvirtError('virDomainMigrateToURI3() failed')
Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused

Steps to reproduce
==================

$ cat queries/libvirt.yaml 
query: >-
  message:"libvirt.libvirtError: unable to connect to server at" AND
  tags:screen-n-cpu.txt

$ elastic-recheck-query queries/libvirt.yaml 
total hits: 12
build_branch
  83% master
  16% stable/ussuri
build_change
  16% 711625
  16% 743231
  16% 754702
  16% 764401
  16% 770547
build_hostids
  16% 13cc9efd4550bb9c731b89fae9c27cc77058360313dea9eebe140cac 1ccf2ea621af4ba5d658324f8716441c624749f7847700ac6aa69490
  16% 322bfced1bb62c11480e9c8cf97e066ab7cb1067b23b38e8004c6f69 23320365a917d59498b30bb8308684708801795984a6ca42b1cd06b7
  16% 422fc847b99ec49a4a48de150fd4c453e34d4b85847603a3435e1511 e1ef7b86aaeb7f61d567417ced5d641b20cff65c4c36e91a6173705f
  16% 77681beb0ddf7a44597e1e8fa6f648861b5edf5445510df6083cdb72 c4902db85fd9c45a1c8b61c8e612d117a8a8f2393128fd1cae285326
  16% 82ed50455b38e6ed7725b8020a977c59f2aaa9db51f018ed12e648b2 3a893be3caac821988438433df3cb21c389ffed93878670819a924f4 24505d497fe40296a37e97a5a1aa1caf33a2c685b883fb60eabf95d5
build_name
  16% devstack-plugin-ceph-multinode-tempest-py3
  16% neutron-ovn-tempest-full-multinode-ovs-master
  16% neutron-tempest-dvr-ha-multinode-full
  16% nova-live-migration
  16% nova-next
build_node
  83% ubuntu-focal
  16% ubuntu-bionic
build_queue
  100% check
build_status
  100% FAILURE
build_zuul_url
  100% N/A
filename
  100% compute1/logs/screen-n-cpu.txt
log_url
  16% https://7b4153b100b6f570d4fc-1ca4271300c9fb8c2811a73921fc79fe.ssl.cf2.rackcdn.com/764401/14/check/neutron-tempest-dvr-ha-multinode-full/d253afd/compute1/logs/screen-n-cpu.txt
  16% https://b0f3338313a78e19953a-f09976335bb1c2233a92ccf49f573637.ssl.cf5.rackcdn.com/754702/5/check/nova-live-migration/76c37d9/compute1/logs/screen-n-cpu.txt
  16% https://cfb2c6e30c49a7b551c1-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf2.rackcdn.com/711625/27/check/devstack-plugin-ceph-multinode-tempest-py3/677c92c/compute1/logs/screen-n-cpu.txt
  16% https://e7591f9adbfbe7dc24c4-cd07d3426af01024dbeaac2f8639d2e0.ssl.cf2.rackcdn.com/743231/9/check/nova-next/cbe27d5/compute1/logs/screen-n-cpu.txt
  16% https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_09a/770547/1/check/tempest-multinode-full-py3/09a5f18/compute1/logs/screen-n-cpu.txt
loglevel
  100% ERROR
module
  100% nova.virt.libvirt.driver
node_provider
  33% ovh-bhs1
  16% airship-kna1
  16% limestone-regionone
  16% rax-ord
  16% vexxhost-ca-ymq-1
port
  16% 39628
  16% 41358
  16% 46144
  16% 52362
  16% 58212
project
  50% openstack/neutron
  33% openstack/nova
  16% openstack/devstack-plugin-ceph
syslog_pid
  16% 19491
  16% 46189
  16% 46724
  16% 52916
  16% 53510
syslog_program
  16% ubuntu-bionic-ovh-bhs1-0022571568 nova-compute
  16% ubuntu-focal-airship-kna1-0022565736 nova-compute
  16% ubuntu-focal-limestone-regionone-0022589731 nova-compute
  16% ubuntu-focal-ovh-bhs1-0022599932 nova-compute
  16% ubuntu-focal-rax-ord-0022553677 nova-compute
tags
  50% screen-n-cpu.txt screen oslofmt
  50% screen-n-cpu.txt screen oslofmt _grokparsefailure
voting
  50% 0
  50% 1
zuul_attempts
  83% 1
  16% 2
zuul_executor
  33% ze06.openstack.org
  33% ze11.openstack.org
  16% ze04.openstack.org
  16% ze09.openstack.org


Expected result
===============
No connection issues with libvirtd.

Actual result
=============
Connection issues with libvirtd cause CI runs to fail.


Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   As above, master and stable/ussuri appear impacted thus far.

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1912310

Title:
  libvirt.libvirtError: unable to connect to server at

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  nova-live-migration fails due to the following error on the src during
  a migration test:

  https://zuul.opendev.org/t/openstack/build/76c37d93ba274cd6b90fbe0659255959/log/compute1/logs/screen-n-cpu.txt

  Jan 18 12:03:25.145915 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation has aborted
  Jan 18 12:03:25.194466 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: INFO nova.compute.manager [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Swapping old allocation on dict_keys(['022fd821-f027-4019-ac4b-cb1e1d2d5423']) held by migration 7b332f03-b32e-4cc8-8e2d-23ed7bee03bd for instance
  Jan 18 12:03:25.239903 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Live Migration failure: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused
  Jan 18 12:03:25.240119 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: DEBUG nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation thread notification {{(pid=52916) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:9448}}
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: Traceback (most recent call last):
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     timer()
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     cb(*args, **kw)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 175, in _do_send
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     waiter.switch(result)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 221, in main
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     result = function(*args, **kwargs)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/utils.py", line 660, in context_wrapper
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     return func(*args, **kwargs)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9103, in _live_migration_operation
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     LOG.error("Live Migration failure: %s", e, instance=instance)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     self.force_reraise()
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     six.reraise(self.type_, self.value, self.tb)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise value
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9091, in _live_migration_operation
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     guest.migrate(self._live_migration_uri(dest),
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 649, in migrate
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     self._domain.migrateToURI3(
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 190, in doit
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 148, in proxy_call
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     rv = execute(f, *args, **kwargs)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 129, in execute
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     six.reraise(c, e, tb)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise value
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 83, in tworker
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     rv = meth(*args, **kwargs)
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:   File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 2099, in migrateToURI3
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]:     raise libvirtError('virDomainMigrateToURI3() failed')
  Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused

  Steps to reproduce
  ==================

  $ cat queries/libvirt.yaml 
  query: >-
    message:"libvirt.libvirtError: unable to connect to server at" AND
    tags:screen-n-cpu.txt

  $ elastic-recheck-query queries/libvirt.yaml 
  total hits: 12
  build_branch
    83% master
    16% stable/ussuri
  build_change
    16% 711625
    16% 743231
    16% 754702
    16% 764401
    16% 770547
  build_hostids
    16% 13cc9efd4550bb9c731b89fae9c27cc77058360313dea9eebe140cac 1ccf2ea621af4ba5d658324f8716441c624749f7847700ac6aa69490
    16% 322bfced1bb62c11480e9c8cf97e066ab7cb1067b23b38e8004c6f69 23320365a917d59498b30bb8308684708801795984a6ca42b1cd06b7
    16% 422fc847b99ec49a4a48de150fd4c453e34d4b85847603a3435e1511 e1ef7b86aaeb7f61d567417ced5d641b20cff65c4c36e91a6173705f
    16% 77681beb0ddf7a44597e1e8fa6f648861b5edf5445510df6083cdb72 c4902db85fd9c45a1c8b61c8e612d117a8a8f2393128fd1cae285326
    16% 82ed50455b38e6ed7725b8020a977c59f2aaa9db51f018ed12e648b2 3a893be3caac821988438433df3cb21c389ffed93878670819a924f4 24505d497fe40296a37e97a5a1aa1caf33a2c685b883fb60eabf95d5
  build_name
    16% devstack-plugin-ceph-multinode-tempest-py3
    16% neutron-ovn-tempest-full-multinode-ovs-master
    16% neutron-tempest-dvr-ha-multinode-full
    16% nova-live-migration
    16% nova-next
  build_node
    83% ubuntu-focal
    16% ubuntu-bionic
  build_queue
    100% check
  build_status
    100% FAILURE
  build_zuul_url
    100% N/A
  filename
    100% compute1/logs/screen-n-cpu.txt
  log_url
    16% https://7b4153b100b6f570d4fc-1ca4271300c9fb8c2811a73921fc79fe.ssl.cf2.rackcdn.com/764401/14/check/neutron-tempest-dvr-ha-multinode-full/d253afd/compute1/logs/screen-n-cpu.txt
    16% https://b0f3338313a78e19953a-f09976335bb1c2233a92ccf49f573637.ssl.cf5.rackcdn.com/754702/5/check/nova-live-migration/76c37d9/compute1/logs/screen-n-cpu.txt
    16% https://cfb2c6e30c49a7b551c1-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf2.rackcdn.com/711625/27/check/devstack-plugin-ceph-multinode-tempest-py3/677c92c/compute1/logs/screen-n-cpu.txt
    16% https://e7591f9adbfbe7dc24c4-cd07d3426af01024dbeaac2f8639d2e0.ssl.cf2.rackcdn.com/743231/9/check/nova-next/cbe27d5/compute1/logs/screen-n-cpu.txt
    16% https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_09a/770547/1/check/tempest-multinode-full-py3/09a5f18/compute1/logs/screen-n-cpu.txt
  loglevel
    100% ERROR
  module
    100% nova.virt.libvirt.driver
  node_provider
    33% ovh-bhs1
    16% airship-kna1
    16% limestone-regionone
    16% rax-ord
    16% vexxhost-ca-ymq-1
  port
    16% 39628
    16% 41358
    16% 46144
    16% 52362
    16% 58212
  project
    50% openstack/neutron
    33% openstack/nova
    16% openstack/devstack-plugin-ceph
  syslog_pid
    16% 19491
    16% 46189
    16% 46724
    16% 52916
    16% 53510
  syslog_program
    16% ubuntu-bionic-ovh-bhs1-0022571568 nova-compute
    16% ubuntu-focal-airship-kna1-0022565736 nova-compute
    16% ubuntu-focal-limestone-regionone-0022589731 nova-compute
    16% ubuntu-focal-ovh-bhs1-0022599932 nova-compute
    16% ubuntu-focal-rax-ord-0022553677 nova-compute
  tags
    50% screen-n-cpu.txt screen oslofmt
    50% screen-n-cpu.txt screen oslofmt _grokparsefailure
  voting
    50% 0
    50% 1
  zuul_attempts
    83% 1
    16% 2
  zuul_executor
    33% ze06.openstack.org
    33% ze11.openstack.org
    16% ze04.openstack.org
    16% ze09.openstack.org

  
  Expected result
  ===============
  No connection issues with libvirtd.

  Actual result
  =============
  Connection issues with libvirtd cause CI runs to fail.

  
  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     As above, master and stable/ussuri appear impacted thus far.

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     libvirt

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1912310/+subscriptions
Follow ups

[Bug 1912310] Re: libvirt.libvirtError: unable to connect to server at
From: OpenStack Infra, 2021-10-07