yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1901383] [NEW] nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: melanie witt <1901383@xxxxxxxxxxxxxxxxxx>
Date: Sun, 25 Oct 2020 07:21:45 -0000
Reply-to: Bug 1901383 <1901383@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Our gate jobs running python 3.6 and libvirt-python 6.8.0 are currently
failing because when we call libvirt methods, the following error is
raised [1]:

Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service [-] Error starting thread.: TypeError: Parameterized generics cannot be used with class or instance checks
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service Traceback (most recent call last):
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 807, in run_service
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     service.start()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.manager.init_host()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1409, in init_host
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 600, in init_host
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._do_quality_warnings()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 577, in _do_quality_warnings
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     caps = self._host.get_capabilities()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 710, in get_capabilities
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 511, in get_connection
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     conn = self._get_connection()
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 494, in _get_connection
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     {'msg': ex})
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.force_reraise()
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise value
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 483, in _get_connection
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 437, in _get_new_connection
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 288, in _connect
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return self._libvirt_proxy.openAuth(uri, auth, flags)
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     result = proxy_call(self._autowrap, f, *args, **kwargs)
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 149, in proxy_call
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     if isinstance(rv, autowrap):
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1162, in __instancecheck__
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return issubclass(instance.__class__, self)
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1148, in __subclasscheck__
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise TypeError("Parameterized generics cannot be used with class "
Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service TypeError: Parameterized generics cannot be used with class or instance checks
Oct 23 12:12:21.459391 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service 


The libvirt-python package version was recently bumped in upper-constraints from version 6.6.0 to 6.8.0:

https://review.opendev.org/750084

and the tempest job on that change ^ passed running python 3.8.

But then our jobs running python 3.6 immediately began to fail.

There appears to be some kind of incompatibility between python 3.6 and
libvirt-python 6.8.0.

I had a look through the libvirt-python repo to see what changed between
6.6.0 and 6.8.0 and can't tell what's the root cause. We get the
TypeError raised in nova-compute when we call the libvirt openAuth
method but nothing in this diff [2] looks problematic?

Not sure what we can do other than revert the upper-constraints bump for
libvirt-python and go back to 6.6.0.

[1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/log/controller/logs/screen-n-cpu.txt?severity=4#861
[2] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d0475c2dfecd386f36307bc70befd9d5bd69daceadc61a5589ee86413d04c7L91

** Affects: nova
     Importance: Critical
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1901383

Title:
  nova-compute fails with "TypeError: Parameterized generics cannot be
  used with class" with python 3.6 and libvirt-python 6.8.0

Status in OpenStack Compute (nova):
  New

Bug description:
  Our gate jobs running python 3.6 and libvirt-python 6.8.0 are
  currently failing because when we call libvirt methods, the following
  error is raised [1]:

  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service [-] Error starting thread.: TypeError: Parameterized generics cannot be used with class or instance checks
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service Traceback (most recent call last):
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 807, in run_service
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     service.start()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.manager.init_host()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1409, in init_host
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 600, in init_host
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._do_quality_warnings()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 577, in _do_quality_warnings
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     caps = self._host.get_capabilities()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 710, in get_capabilities
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 511, in get_connection
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     conn = self._get_connection()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 494, in _get_connection
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     {'msg': ex})
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.force_reraise()
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise value
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 483, in _get_connection
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 437, in _get_new_connection
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 288, in _connect
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return self._libvirt_proxy.openAuth(uri, auth, flags)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 149, in proxy_call
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     if isinstance(rv, autowrap):
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1162, in __instancecheck__
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return issubclass(instance.__class__, self)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1148, in __subclasscheck__
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise TypeError("Parameterized generics cannot be used with class "
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service TypeError: Parameterized generics cannot be used with class or instance checks
  Oct 23 12:12:21.459391 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service 

  
  The libvirt-python package version was recently bumped in upper-constraints from version 6.6.0 to 6.8.0:

  https://review.opendev.org/750084

  and the tempest job on that change ^ passed running python 3.8.

  But then our jobs running python 3.6 immediately began to fail.

  There appears to be some kind of incompatibility between python 3.6
  and libvirt-python 6.8.0.

  I had a look through the libvirt-python repo to see what changed
  between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get
  the TypeError raised in nova-compute when we call the libvirt openAuth
  method but nothing in this diff [2] looks problematic?

  Not sure what we can do other than revert the upper-constraints bump
  for libvirt-python and go back to 6.6.0.

  [1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/log/controller/logs/screen-n-cpu.txt?severity=4#861
  [2] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d0475c2dfecd386f36307bc70befd9d5bd69daceadc61a5589ee86413d04c7L91

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1901383/+subscriptions
Follow ups

[Bug 1901383] Re: nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0
From: Elod Illes, 2021-04-13
[Bug 1901383] Re: nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0
From: Elod Illes, 2020-11-20
[Bug 1901383] Re: nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0
From: melanie witt, 2020-11-16
[Bug 1901383] Re: nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0
From: OpenStack Infra, 2020-10-28