yahoo-eng-team team mailing list archive

[melanie witt <2091033@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

tl;dr This bug has the same root cause as
https://bugs.launchpad.net/nova/+bug/1840912 where items in lists
returned from libvirt are not automatically wrapped in a tpool.Proxy.

Discovered during investigation of a downstream bug [1] where a live
migration dirtying memory faster than the transfer and nova-compute
became frozen unable to perform any other operations, not even logging,
for hours.

The freezing was tracked down to un-proxied libvirt call
listAllDevices() which could block all other greenthreads. The
listAllDevices() call occurs during the update_available_resource()
periodic task in the libvirt driver in _get_pci_passthrough_devices().
In a GMR collected during a repro of the issue, a traceback showing this
was present in the report [2]:

tderr F /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:222 in run_periodic_tasks
stderr F     `task(self, context)`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/compute/manager.py:9142 in update_available_resource
stderr F     `startup=startup)`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/compute/manager.py:9056 in _update_available_resource_for_node
stderr F     `startup=startup)`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py:911 in update_available_resource
stderr F     `resources = self.driver.get_available_resource(nodename)`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:8369 in get_available_resource
stderr F     `data['pci_passthrough_devices'] = self._get_pci_passthrough_devices()`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:7080 in _get_pci_passthrough_devices
stderr F     `in devices.items() if "pci" in dev.listCaps()]`
stderr F 
stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:7080 in <listcomp>
stderr F     `in devices.items() if "pci" in dev.listCaps()]`
stderr F 
stderr F /usr/lib64/python3.6/site-packages/libvirt.py:6313 in listCaps
stderr F     `ret = libvirtmod.virNodeDeviceListCaps(self._o)`

The listAllDevices() function returned a list of unwrapped virNodeDevice
objects and so calling listCaps() on such an unwrapped device could
cause a freeze.

Based on the above, the bug reporter was able to test a patch [3] to
wrap listAllDevices() list items in tpool.Proxy and the result showed
nova-compute no longer freezing [4] in the aforementioned scenario.

During investigation it was also noticed that the listDevices() call
list items were not tpool.Proxy wrapped, so this is fixed as well in the
patch.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=2312196
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2312196#c13
[3] https://review.opendev.org/c/openstack/nova/+/932669
[4] https://bugzilla.redhat.com/show_bug.cgi?id=2312196#c21

** Affects: nova
     Importance: Undecided
     Assignee: melanie witt (melwitt)
         Status: In Progress


** Tags: compute libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2091033

Title:
  Un-proxied libvirt calls list(All)Devices() can cause nova-compute to
  freeze for hours

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  tl;dr This bug has the same root cause as
  https://bugs.launchpad.net/nova/+bug/1840912 where items in lists
  returned from libvirt are not automatically wrapped in a tpool.Proxy.

  Discovered during investigation of a downstream bug [1] where a live
  migration dirtying memory faster than the transfer and nova-compute
  became frozen unable to perform any other operations, not even
  logging, for hours.

  The freezing was tracked down to un-proxied libvirt call
  listAllDevices() which could block all other greenthreads. The
  listAllDevices() call occurs during the update_available_resource()
  periodic task in the libvirt driver in _get_pci_passthrough_devices().
  In a GMR collected during a repro of the issue, a traceback showing
  this was present in the report [2]:

  tderr F /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:222 in run_periodic_tasks
  stderr F     `task(self, context)`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/compute/manager.py:9142 in update_available_resource
  stderr F     `startup=startup)`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/compute/manager.py:9056 in _update_available_resource_for_node
  stderr F     `startup=startup)`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py:911 in update_available_resource
  stderr F     `resources = self.driver.get_available_resource(nodename)`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:8369 in get_available_resource
  stderr F     `data['pci_passthrough_devices'] = self._get_pci_passthrough_devices()`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:7080 in _get_pci_passthrough_devices
  stderr F     `in devices.items() if "pci" in dev.listCaps()]`
  stderr F 
  stderr F /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:7080 in <listcomp>
  stderr F     `in devices.items() if "pci" in dev.listCaps()]`
  stderr F 
  stderr F /usr/lib64/python3.6/site-packages/libvirt.py:6313 in listCaps
  stderr F     `ret = libvirtmod.virNodeDeviceListCaps(self._o)`

  The listAllDevices() function returned a list of unwrapped
  virNodeDevice objects and so calling listCaps() on such an unwrapped
  device could cause a freeze.

  Based on the above, the bug reporter was able to test a patch [3] to
  wrap listAllDevices() list items in tpool.Proxy and the result showed
  nova-compute no longer freezing [4] in the aforementioned scenario.

  During investigation it was also noticed that the listDevices() call
  list items were not tpool.Proxy wrapped, so this is fixed as well in
  the patch.

  
  [1] https://bugzilla.redhat.com/show_bug.cgi?id=2312196
  [2] https://bugzilla.redhat.com/show_bug.cgi?id=2312196#c13
  [3] https://review.opendev.org/c/openstack/nova/+/932669
  [4] https://bugzilla.redhat.com/show_bug.cgi?id=2312196#c21

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2091033/+subscriptions