yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1646779] Re: Cannot connect to libvirt

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrea Frittoli <1646779@xxxxxxxxxxxxxxxxxx>
Date: Wed, 07 Dec 2016 16:57:35 -0000
Reply-to: Bug 1646779 <1646779@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1646779

Title:
  Cannot connect to libvirt

Status in OpenStack Compute (nova):
  New
Status in tempest:
  New

Bug description:
  A VM fails to spawn with no host available. The nova-cpu logs reveals
  a problem connecting to libvirt. 84 hits since Nov 23rd:

  message: "libvirtError: Failed to connect socket to '/var/run/libvirt
  /libvirt-sock': Connection refused"

  Recent failure: http://logs.openstack.org/66/401366/4/gate/gate-
  tempest-dsvm-neutron-full-ubuntu-
  xenial/3deacc5/logs/screen-n-cpu.txt.gz?level=ERROR

  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host [req-12fbb338-7df0-4654-b686-257245421442 tempest-ImagesOneServerNegativeTestJSON-1400886372 tempest-ImagesOneServerNegativeTestJSON-1400886372] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host Traceback (most recent call last):
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 453, in get_connection
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     conn = self._get_connection()
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 436, in _get_connection
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     {'msg': ex})
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     self.force_reraise()
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     six.reraise(self.type_, self.value, self.tb)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 425, in _get_connection
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     self._wrapped_conn = self._get_new_connection()
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 370, in _get_new_connection
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     wrapped_conn = self._connect(self._uri, self._read_only)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 226, in _connect
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     libvirt.openAuth, uri, auth, flags)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     rv = execute(f, *args, **kwargs)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     six.reraise(c, e, tb)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     rv = meth(*args, **kwargs)
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
  2016-12-01 18:16:05.117 6160 ERROR nova.virt.libvirt.host 
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [req-12fbb338-7df0-4654-b686-257245421442 tempest-ImagesOneServerNegativeTestJSON-1400886372 tempest-ImagesOneServerNegativeTestJSON-1400886372] [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc] Instance failed to spawn
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc] Traceback (most recent call last):
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2117, in _build_resources
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     yield resources
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1924, in _build_and_run_instance
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     block_device_info=block_device_info)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2647, in spawn
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     block_device_info=block_device_info)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4741, in _get_guest_xml
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     context)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4515, in _get_guest_config
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     instance.numa_topology, flavor, allowed_cpus, image_meta)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3872, in _get_guest_numa_config
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     if (not self._has_numa_support() and
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5212, in _has_numa_support
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     if self._host.has_version(ver):
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 519, in has_version
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     lv_ver=lv_ver, hv_ver=hv_ver, hv_type=hv_type, op=operator.ne)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 490, in _version_check
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     conn = self.get_connection()
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 462, in get_connection
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc]     raise exception.HypervisorUnavailable(host=CONF.host)
  2016-12-01 18:16:05.123 6160 ERROR nova.compute.manager [instance: 6fa73b04-c6a7-47a8-908b-6738f36f6ffc] HypervisorUnavailable: Connection to the hypervisor is broken on host: ubuntu-xenial-rax-ord-5924789

  
  The issue happens with different tests, so I doubt it is actually a Tempest issue, but I'm filing the issue here for initial triage.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1646779/+subscriptions