yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1782696] [NEW] nova migration a vm with pci devices to a host don't have pci available

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yongli He <yongli.he@xxxxxxxxx>
Date: Fri, 20 Jul 2018 05:39:58 -0000
Reply-to: Bug 1782696 <1782696@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

on nova master, multinodes with 2 compute nodes available. only one of
nova compute had pci devices available.

1. booting vm with pci reqested, assume it's name: pci-vm
2. nova migrate pci-vm
3. the pci-vm been schedule to a host don't have pci devices available.

we see exception on nova compute(the host don't have pci devs and which
nova schedule the pci-vm to):

2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server [req-71f0427e-4bd3-4014-8807-179e6e718066 b7d4ff5b9c0b4739865cce43a8de1c64 f0584549d87143dfbf235199c653f33c - default default] Exception during message handling: PciRequestAliasNotDefined: PCI alias intelpci is not defined
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 185, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 155, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/utils.py", line 1090, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 213, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 201, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4181, in prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     filter_properties, host_list)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4245, in _reschedule_resize_or_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(*exc_info)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4160, in prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     node, migration, clean_shutdown)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4120, in _prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits) as claim:
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 254, in resize_claim
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 310, in _move_claim
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     new_instance_type)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", line 219, in get_pci_requests_from_flavor
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     flavor['extra_specs']['pci_passthrough:alias'])
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", line 157, in _translate_alias_to_requests
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     raise exception.PciRequestAliasNotDefined(alias=name)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server PciRequestAliasNotDefined: PCI alias intelpci is not defined

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1782696

Title:
  nova migration a vm with pci devices to a host don't have pci
  available

Status in OpenStack Compute (nova):
  New

Bug description:
  on nova master, multinodes with 2 compute nodes available. only one of
  nova compute had pci devices available.

  1. booting vm with pci reqested, assume it's name: pci-vm
  2. nova migrate pci-vm
  3. the pci-vm been schedule to a host don't have pci devices available.

  we see exception on nova compute(the host don't have pci devs and
  which nova schedule the pci-vm to):

  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server [req-71f0427e-4bd3-4014-8807-179e6e718066 b7d4ff5b9c0b4739865cce43a8de1c64 f0584549d87143dfbf235199c653f33c - default default] Exception during message handling: PciRequestAliasNotDefined: PCI alias intelpci is not defined
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 185, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 155, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/utils.py", line 1090, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 213, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 201, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4181, in prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     filter_properties, host_list)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4245, in _reschedule_resize_or_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     six.reraise(*exc_info)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4160, in prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     node, migration, clean_shutdown)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4120, in _prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits) as claim:
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 254, in resize_claim
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 310, in _move_claim
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     new_instance_type)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", line 219, in get_pci_requests_from_flavor
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     flavor['extra_specs']['pci_passthrough:alias'])
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", line 157, in _translate_alias_to_requests
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     raise exception.PciRequestAliasNotDefined(alias=name)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server PciRequestAliasNotDefined: PCI alias intelpci is not defined

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1782696/+subscriptions
Follow ups

[Bug 1782696] Re: nova migration a vm with pci devices to a host don't have pci available
From: sean mooney, 2020-06-29
[Bug 1782696] Re: nova migration a vm with pci devices to a host don't have pci available
From: Ivan Borisov, 2020-05-14
[Bug 1782696] Re: nova migration a vm with pci devices to a host don't have pci available
From: Matt Riedemann, 2019-06-20