yahoo-eng-team team mailing list archive

[guolei <1901170@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
When we boot a GPU instance, on nova-compute's instance_claim
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/resource_tracker.py#L197-L217
input instance object's attribute 'pci_devices'  had update from [] to [PciDevice], it include a calculated GPU PCI device object.

Ok, now we pay attention to claim's code flow:
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/claims.py#L64
it cloned input instance object, set to self.instance
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/claims.py#L78-L84
abort func will abort instance's claim with self.instance, it a cloned one, not the origin input instance object.

Now, we can see, if spawn instance failed, claim.abort will be called, it revert the cloned instance object's
 'pci_devices' attribute to [], and pci_device in db had reverted from allocate to free too. but the origin input instance object not, origin instance object's 'pci_devices' is still [PciDevice], and it will send to nova-conductor to do reschedule, and on next node, after claim, instance.pci_devices will be [PciDevice, PciDevice]

And then, spawn instance will have two GPU pci device, or raise a
LibvirtError, "Device xxx is in used"

Steps to reproduce
==================
1. build libvirt error on all compute nodes 
2. nova boot a GPU instance
3. show guest xml in nova-compute.log

Expected result
===============
on reschedule node, guest xml had just one GPU pci device

Actual result
=============
on reschedule node, guest xml had more then one GPU pci device

** Affects: nova
     Importance: Undecided
     Assignee: guolei (guolei-5)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => guolei (guolei-5)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1901170

Title:
  nova boot GPU instance will attach more one GPU pci device when
  reschedule happened

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When we boot a GPU instance, on nova-compute's instance_claim
  https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/resource_tracker.py#L197-L217
  input instance object's attribute 'pci_devices'  had update from [] to [PciDevice], it include a calculated GPU PCI device object.

  Ok, now we pay attention to claim's code flow:
  https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/claims.py#L64
  it cloned input instance object, set to self.instance
  https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/claims.py#L78-L84
  abort func will abort instance's claim with self.instance, it a cloned one, not the origin input instance object.

  Now, we can see, if spawn instance failed, claim.abort will be called, it revert the cloned instance object's
   'pci_devices' attribute to [], and pci_device in db had reverted from allocate to free too. but the origin input instance object not, origin instance object's 'pci_devices' is still [PciDevice], and it will send to nova-conductor to do reschedule, and on next node, after claim, instance.pci_devices will be [PciDevice, PciDevice]

  And then, spawn instance will have two GPU pci device, or raise a
  LibvirtError, "Device xxx is in used"

  Steps to reproduce
  ==================
  1. build libvirt error on all compute nodes 
  2. nova boot a GPU instance
  3. show guest xml in nova-compute.log

  Expected result
  ===============
  on reschedule node, guest xml had just one GPU pci device

  Actual result
  =============
  on reschedule node, guest xml had more then one GPU pci device

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1901170/+subscriptions