yahoo-eng-team team mailing list archive

[OpenStack Infra <1901383@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/759831
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d2ca53bb86b8e4a3c44855cb5ef57f223462543
Submitter: Jens Harbott (frickler) (j.harbott@xxxxxxxx)
Branch:    master

commit 0d2ca53bb86b8e4a3c44855cb5ef57f223462543
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date:   Tue Oct 27 08:44:59 2020 +0000

    libvirt: Only ask tpool.Proxy to autowrap vir* classes
    
    I668643c836d46a25df46d4c99a973af5e50a39db attempted to fix service wide
    pauses by providing a more complete list of classes to tpool.Proxy.
    
    While this excluded libvirtError it can include internal libvirt-python
    classes pointed to by private globals that have been introduced with the
    use of type checking within the module.
    
    Any attempt to wrap these internal classes will result in the failure
    seen in bug #1901383. As a result this change simply ignores any class
    found during inspection that doesn't start with the `vir` string, used
    by libvirt to denote public methods and classes.
    
    Closes-Bug: #1901383
    Co-Authored-By: Daniel Berrange <berrange@xxxxxxxxxx>
    Change-Id: I568b0c4fd6069b9118ff116532f14abb46cc42ab


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1901383

Title:
  nova-compute fails with "TypeError: Parameterized generics cannot be
  used with class" with python 3.6 and libvirt-python 6.8.0

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Our gate jobs running python 3.6 and libvirt-python 6.8.0 are
  currently failing because when we call libvirt methods, the following
  error is raised [1]:

  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service [-] Error starting thread.: TypeError: Parameterized generics cannot be used with class or instance checks
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service Traceback (most recent call last):
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 807, in run_service
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     service.start()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 159, in start
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.manager.init_host()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1409, in init_host
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 600, in init_host
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._do_quality_warnings()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 577, in _do_quality_warnings
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     caps = self._host.get_capabilities()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 710, in get_capabilities
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 511, in get_connection
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     conn = self._get_connection()
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 494, in _get_connection
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     {'msg': ex})
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self.force_reraise()
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     six.reraise(self.type_, self.value, self.tb)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise value
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 483, in _get_connection
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     self._wrapped_conn = self._get_new_connection()
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 437, in _get_new_connection
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     wrapped_conn = self._connect(self._uri, self._read_only)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/host.py", line 288, in _connect
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return self._libvirt_proxy.openAuth(uri, auth, flags)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 149, in proxy_call
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     if isinstance(rv, autowrap):
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1162, in __instancecheck__
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     return issubclass(instance.__class__, self)
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service   File "/usr/lib/python3.6/typing.py", line 1148, in __subclasscheck__
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service     raise TypeError("Parameterized generics cannot be used with class "
  Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service TypeError: Parameterized generics cannot be used with class or instance checks
  Oct 23 12:12:21.459391 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service 

  
  The libvirt-python package version was recently bumped in upper-constraints from version 6.6.0 to 6.8.0:

  https://review.opendev.org/750084

  and the tempest job on that change ^ passed running python 3.8.

  But then our jobs running python 3.6 immediately began to fail.

  There appears to be some kind of incompatibility between python 3.6
  and libvirt-python 6.8.0.

  I had a look through the libvirt-python repo to see what changed
  between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get
  the TypeError raised in nova-compute when we call the libvirt openAuth
  method but nothing in this diff [2] looks problematic?

  Not sure what we can do other than revert the upper-constraints bump
  for libvirt-python and go back to 6.6.0.

  [1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/log/controller/logs/screen-n-cpu.txt?severity=4#861
  [2] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d0475c2dfecd386f36307bc70befd9d5bd69daceadc61a5589ee86413d04c7L91

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1901383/+subscriptions