yahoo-eng-team team mailing list archive

[OpenStack Infra <2115729@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/953972
Committed: https://opendev.org/openstack/nova/commit/0208be629c3853863bcd49b8bdbe2b9889b85012
Submitter: "Zuul (22348)"
Branch:    master

commit 0208be629c3853863bcd49b8bdbe2b9889b85012
Author: Balazs Gibizer <gibi@xxxxxxxxxx>
Date:   Wed Jul 2 18:01:41 2025 +0200

    Fix pci_tracker.save to delete all removed devs
    
    The bug is that the save code modifies the self.pci_devs.object list
    while iterating it. This resulted in only half of the removed devs was
    deleted at each save() run. Iterating on a shallow copy of that list
    fixes it.
    
    Closes-Bug: #2115729
    Change-Id: I2711be9605618c1b93104d1dbddd8c7ee73b577e
    Signed-off-by: Balazs Gibizer <gibi@xxxxxxxxxx>


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2115729

Title:
  Removing a lot of PCI devices via changing device_spec seems to be
  slow

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  stack@aio:~$ grep device_spec /etc/nova/nova-cpu.conf 
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:06:", "physical_network": "physnet2"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:07:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:08:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:09:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:0a:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:0b:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0c:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0d:"}
  device_spec = {}
  stack@aio:~$ mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  80
  stack@aio:~$ grep device_spec /etc/nova/nova-cpu.conf 
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:06:", "physical_network": "physnet2"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:07:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:08:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:09:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:0a:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10ca", "address": "0000:0b:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0c:"}
  #device_spec = { "vendor_id":"8086", "product_id":"10c9", "address": "0000:0d:"}
  #device_spec = {}
  stack@aio:~$ sudo systemctl restart devstack@n-cpu
  stack@aio:~$ mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  58
  stack@aio:~$ date && mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  Tue Jul  1 12:42:13 UTC 2025
  40
  stack@aio:~$ date && mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  Tue Jul  1 12:42:31 UTC 2025
  40
  stack@aio:~$ date && mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  Tue Jul  1 12:43:09 UTC 2025
  20
  stack@aio:~$ date && mysql -e 'select deleted, address from nova_cell1.pci_devices where compute_node_id = 1 and deleted = 0;' | wc -l
  Tue Jul  1 12:50:35 UTC 2025
  0

  It feels like nova-compute at each periodic run deletes half of the
  removed deviced only. But eventually removes them all.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2115729/+subscriptions