yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622854] [NEW] pci: double pci migration is putting vm in ERROR

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Moshe Levi <moshele@xxxxxxxxxxxx>
Date: Tue, 13 Sep 2016 06:33:06 -0000
Reply-to: Bug 1622854 <1622854@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

nova master
devstack multinode with 2 compute nodes
1. booting vm with direct port
2. nova migrate 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
3. nova resize-confirm 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
4. nova migrate 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
5. nova resize-confirm 128a2ba4-fb6e-49f4-a6e0-45cde1c60215

The second migration failed with this error:

2016-09-12 13:12:45.750 8388 DEBUG oslo_concurrency.lockutils [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.143s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Error updating resources for node r-dcs224.mtr.labs.mlnx.
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager Traceback (most recent call last):
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/manager.py", line 6408, in update_available_resource_for_node
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     rt.update_available_resource(context)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/resource_tracker.py", line 526, in update_available_resource
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     return f(*args, **kwargs)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/resource_tracker.py", line 580, in _update_available_resource
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self.pci_tracker.clean_usage(instances, migrations, orphans)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/pci/manager.py", line 326, in clean_usage
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self._free_device(dev)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/pci/manager.py", line 270, in _free_device
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     freed_devs = dev.free(instance)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/objects/pci_device.py", line 397, in free
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     hopestatus=ok_statuses)
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager PciDeviceInvalidStatus: PCI device 3:0000:03:00.5 is available instead of ('allocated', 'claimed')
2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager
2016-09-12 13:12:46.220 8388 DEBUG oslo_service.periodic_task [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Running periodic task ComputeManager._sync_scheduler_instance_info run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1622854

Title:
  pci: double pci migration is putting vm in ERROR

Status in OpenStack Compute (nova):
  New

Bug description:
  nova master
  devstack multinode with 2 compute nodes
  1. booting vm with direct port
  2. nova migrate 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
  3. nova resize-confirm 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
  4. nova migrate 128a2ba4-fb6e-49f4-a6e0-45cde1c60215
  5. nova resize-confirm 128a2ba4-fb6e-49f4-a6e0-45cde1c60215

  The second migration failed with this error:

  2016-09-12 13:12:45.750 8388 DEBUG oslo_concurrency.lockutils [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.143s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Error updating resources for node r-dcs224.mtr.labs.mlnx.
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager Traceback (most recent call last):
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/manager.py", line 6408, in update_available_resource_for_node
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     rt.update_available_resource(context)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/resource_tracker.py", line 526, in update_available_resource
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     return f(*args, **kwargs)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/compute/resource_tracker.py", line 580, in _update_available_resource
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self.pci_tracker.clean_usage(instances, migrations, orphans)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/pci/manager.py", line 326, in clean_usage
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     self._free_device(dev)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/pci/manager.py", line 270, in _free_device
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     freed_devs = dev.free(instance)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager   File "/.autodirect/mtrswgwork/moshele/openstack/nova/nova/objects/pci_device.py", line 397, in free
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager     hopestatus=ok_statuses)
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager PciDeviceInvalidStatus: PCI device 3:0000:03:00.5 is available instead of ('allocated', 'claimed')
  2016-09-12 13:12:45.750 8388 ERROR nova.compute.manager
  2016-09-12 13:12:46.220 8388 DEBUG oslo_service.periodic_task [req-a4a0126a-215a-489a-b043-ad38d3b5e28d - -] Running periodic task ComputeManager._sync_scheduler_instance_info run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1622854/+subscriptions
Follow ups

[Bug 1622854] Re: pci: double pci migration is putting vm in ERROR
From: OpenStack Infra, 2016-11-03