yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1822357] [NEW] nova-compute don't start in grenade job due to missing libvirt module

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1822357@xxxxxxxxxxxxxxxxxx>
Date: Fri, 29 Mar 2019 18:12:12 -0000
Reply-to: Bug 1822357 <1822357@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I found that sometimes grenade jobs are failing and reason of that failure is that nova-compute is not started.
Error from nova-compute log:

Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver [None req-8a6cbe05-4b28-49b2-866b-9a35b1eb785d None None] Unable to load the virtualization driver: ModuleNotFoundError: No module named 'libvirt'
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver Traceback (most recent call last):
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/opt/stack/old/nova/nova/virt/driver.py", line 1747, in load_compute_driver
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     virtapi)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 44, in import_object
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     return import_class(import_str)(*args, **kwargs)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 320, in __init__
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     libvirt = importutils.import_module('libvirt')
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 73, in import_module
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     __import__(import_str)
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver ModuleNotFoundError: No module named 'libvirt'
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver 
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]:   """)
Mar 29 14:04:39.264259 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE
Mar 29 14:04:39.264289 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'.

It is here: http://logs.openstack.org/45/638645/11/gate/neutron-grenade-
dvr-multinode/4064577/logs/screen-n-cpu.txt.gz

Logstash query which I used to find such examples:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'libvirt'%5C%22

It happened more than 10 times in last 7 days so it is quite frequent
issue.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1822357

Title:
  nova-compute don't start in grenade job due to missing libvirt module

Status in OpenStack Compute (nova):
  New

Bug description:
  I found that sometimes grenade jobs are failing and reason of that failure is that nova-compute is not started.
  Error from nova-compute log:

  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver [None req-8a6cbe05-4b28-49b2-866b-9a35b1eb785d None None] Unable to load the virtualization driver: ModuleNotFoundError: No module named 'libvirt'
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver Traceback (most recent call last):
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/opt/stack/old/nova/nova/virt/driver.py", line 1747, in load_compute_driver
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     virtapi)
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 44, in import_object
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     return import_class(import_str)(*args, **kwargs)
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 320, in __init__
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     libvirt = importutils.import_module('libvirt')
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/importutils.py", line 73, in import_module
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver     __import__(import_str)
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver ModuleNotFoundError: No module named 'libvirt'
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: ERROR nova.virt.driver 
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]: /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  Mar 29 14:04:38.720553 ubuntu-bionic-rax-dfw-0004533915 nova-compute[29886]:   """)
  Mar 29 14:04:39.264259 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE
  Mar 29 14:04:39.264289 ubuntu-bionic-rax-dfw-0004533915 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'.

  It is here: http://logs.openstack.org/45/638645/11/gate/neutron-
  grenade-dvr-multinode/4064577/logs/screen-n-cpu.txt.gz

  Logstash query which I used to find such examples:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'libvirt'%5C%22

  It happened more than 10 times in last 7 days so it is quite frequent
  issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1822357/+subscriptions
Follow ups

[Bug 1822357] Re: nova-compute don't start in grenade job due to missing libvirt module
From: Matt Riedemann, 2019-04-01