yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1763382] [NEW] "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" in nova-multiattach job since we started using the Queens UCA

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 12 Apr 2018 13:46:30 -0000
Reply-to: Bug 1763382 <1763382@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Since the devstack patch to start using the queens UCA merged on master
(rocky):

https://review.openstack.org/#/c/554314/

The nova-multiattach job has been failing with this:

http://logs.openstack.org/49/560349/1/check/nova-
multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR#_Apr_12_07_23_19_791688

Apr 12 07:23:19.791688 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host [None req-fb02f74a-6ce9-4aa6-abc4-24a6d87ca6b0 None None] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
Apr 12 07:23:19.791867 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host Traceback (most recent call last):
Apr 12 07:23:19.792029 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 443, in get_connection
Apr 12 07:23:19.792186 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     conn = self._get_connection()
Apr 12 07:23:19.792363 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 426, in _get_connection
Apr 12 07:23:19.792521 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     {'msg': ex})
Apr 12 07:23:19.792693 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 12 07:23:19.792847 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     self.force_reraise()
Apr 12 07:23:19.793001 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 12 07:23:19.793171 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     six.reraise(self.type_, self.value, self.tb)
Apr 12 07:23:19.793329 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 415, in _get_connection
Apr 12 07:23:19.793487 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     self._wrapped_conn = self._get_new_connection()
Apr 12 07:23:19.793664 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 369, in _get_new_connection
Apr 12 07:23:19.793821 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     wrapped_conn = self._connect(self._uri, self._read_only)
Apr 12 07:23:19.794036 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 225, in _connect
Apr 12 07:23:19.794191 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     libvirt.openAuth, uri, auth, flags)
Apr 12 07:23:19.794346 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Apr 12 07:23:19.794512 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     rv = execute(f, *args, **kwargs)
Apr 12 07:23:19.794666 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 12 07:23:19.794823 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     six.reraise(c, e, tb)
Apr 12 07:23:19.794995 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 12 07:23:19.795160 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     rv = meth(*args, **kwargs)
Apr 12 07:23:19.795315 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
Apr 12 07:23:19.795470 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
Apr 12 07:23:19.795626 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
Apr 12 07:23:19.795788 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host 

There is a patch to make the nova-multiattach job using the Queens UCA,
which was passing before this, but now is also hitting that issue:

https://review.openstack.org/#/c/554317/

http://logs.openstack.org/17/554317/3/check/nova-
multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR

My guess would be it has something to do with the libvirt group?

https://review.openstack.org/#/c/554314/1/stackrc

Anyway, we can make the nova-multiattach job non-voting until we get
this sorted out.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: gate-failure libvirt multiattach volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1763382

Title:
  "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-
  sock': Permission denied" in nova-multiattach job since we started
  using the Queens UCA

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Since the devstack patch to start using the queens UCA merged on
  master (rocky):

  https://review.openstack.org/#/c/554314/

  The nova-multiattach job has been failing with this:

  http://logs.openstack.org/49/560349/1/check/nova-
  multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR#_Apr_12_07_23_19_791688

  Apr 12 07:23:19.791688 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host [None req-fb02f74a-6ce9-4aa6-abc4-24a6d87ca6b0 None None] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
  Apr 12 07:23:19.791867 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host Traceback (most recent call last):
  Apr 12 07:23:19.792029 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 443, in get_connection
  Apr 12 07:23:19.792186 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     conn = self._get_connection()
  Apr 12 07:23:19.792363 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 426, in _get_connection
  Apr 12 07:23:19.792521 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     {'msg': ex})
  Apr 12 07:23:19.792693 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Apr 12 07:23:19.792847 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     self.force_reraise()
  Apr 12 07:23:19.793001 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Apr 12 07:23:19.793171 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     six.reraise(self.type_, self.value, self.tb)
  Apr 12 07:23:19.793329 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 415, in _get_connection
  Apr 12 07:23:19.793487 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     self._wrapped_conn = self._get_new_connection()
  Apr 12 07:23:19.793664 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 369, in _get_new_connection
  Apr 12 07:23:19.793821 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     wrapped_conn = self._connect(self._uri, self._read_only)
  Apr 12 07:23:19.794036 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 225, in _connect
  Apr 12 07:23:19.794191 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     libvirt.openAuth, uri, auth, flags)
  Apr 12 07:23:19.794346 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  Apr 12 07:23:19.794512 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     rv = execute(f, *args, **kwargs)
  Apr 12 07:23:19.794666 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  Apr 12 07:23:19.794823 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     six.reraise(c, e, tb)
  Apr 12 07:23:19.794995 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  Apr 12 07:23:19.795160 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     rv = meth(*args, **kwargs)
  Apr 12 07:23:19.795315 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
  Apr 12 07:23:19.795470 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
  Apr 12 07:23:19.795626 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
  Apr 12 07:23:19.795788 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host 

  There is a patch to make the nova-multiattach job using the Queens
  UCA, which was passing before this, but now is also hitting that
  issue:

  https://review.openstack.org/#/c/554317/

  http://logs.openstack.org/17/554317/3/check/nova-
  multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR

  My guess would be it has something to do with the libvirt group?

  https://review.openstack.org/#/c/554314/1/stackrc

  Anyway, we can make the nova-multiattach job non-voting until we get
  this sorted out.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1763382/+subscriptions
Follow ups

[Bug 1763382] Re: "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" in nova-multiattach job since we started using the Queens UCA
From: OpenStack Infra, 2018-04-19