← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #84935

[Bug 1912273] [NEW] SRIOV instance Error: Exception during message handling: KeyError: 'pci_slot'

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I am running stein version of openstack and running SR-IOV instance but
today vm die with some memory issue (might be oom) so i start to re-
start using following command but got error on compute node and failed
to start.

nova start <vm-name>

Error on compute nova logs, this is happened to multiple VMs on
different compute nodes.

---------

2021-01-18 17:18:31.396 1496 INFO nova.virt.libvirt.imagecache [req-d82a76a2-edd6-41d7-a7be-7245997ac4b1 - - - - -] image c5d0e40e-3b5a-44a0-9ac3-e6e59b3c6276 at (/var/lib/nova/instances/_base/b62c49c5d32aafa9028bd2b518699eab17dd07e0): checking
2021-01-18 17:18:31.638 1496 INFO nova.virt.libvirt.imagecache [req-d82a76a2-edd6-41d7-a7be-7245997ac4b1 - - - - -] Active base files: /var/lib/nova/instances/_base/b62c49c5d32aafa9028bd2b518699eab17dd07e0
2021-01-18 17:19:04.007 1496 INFO nova.virt.libvirt.driver [-] [instance: 041ba2fe-2a0f-4918-820f-e6401ea0255b] Instance destroyed successfully.
2021-01-18 17:19:04.640 1496 INFO nova.compute.manager [req-ac9ab44c-c6bc-421c-8c6d-7a777cb4575e 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] [instance: 041ba2fe-2a0f-4918-820f-e6401ea0255b] Successfully reverted task state from powering-on on failure for instance.
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server [req-ac9ab44c-c6bc-421c-8c6d-7a777cb4575e 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Exception during message handling: KeyError: 'pci_slot'
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/utils.py", line 1323, in decorated_function
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 2922, in start_instance
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 2892, in _power_on
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     block_device_info)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2987, in power_on
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2855, in _hard_reboot
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     mdevs=mdevs)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5482, in _get_guest_xml
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     context, mdevs)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5282, in _get_guest_config
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     flavor, virt_type, self._host)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 608, in get_config
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     inst_type, virt_type, host)
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 353, in get_config_hw_veb
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     conf, net_type, profile['pci_slot'],
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server

---------

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1912273

Title:
  SRIOV instance Error: Exception during message handling: KeyError:
  'pci_slot'

Status in OpenStack Compute (nova):
  New

Bug description:
  I am running stein version of openstack and running SR-IOV instance
  but today vm die with some memory issue (might be oom) so i start to
  re-start using following command but got error on compute node and
  failed to start.

  nova start <vm-name>

  Error on compute nova logs, this is happened to multiple VMs on
  different compute nodes.

  ---------

  2021-01-18 17:18:31.396 1496 INFO nova.virt.libvirt.imagecache [req-d82a76a2-edd6-41d7-a7be-7245997ac4b1 - - - - -] image c5d0e40e-3b5a-44a0-9ac3-e6e59b3c6276 at (/var/lib/nova/instances/_base/b62c49c5d32aafa9028bd2b518699eab17dd07e0): checking
  2021-01-18 17:18:31.638 1496 INFO nova.virt.libvirt.imagecache [req-d82a76a2-edd6-41d7-a7be-7245997ac4b1 - - - - -] Active base files: /var/lib/nova/instances/_base/b62c49c5d32aafa9028bd2b518699eab17dd07e0
  2021-01-18 17:19:04.007 1496 INFO nova.virt.libvirt.driver [-] [instance: 041ba2fe-2a0f-4918-820f-e6401ea0255b] Instance destroyed successfully.
  2021-01-18 17:19:04.640 1496 INFO nova.compute.manager [req-ac9ab44c-c6bc-421c-8c6d-7a777cb4575e 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] [instance: 041ba2fe-2a0f-4918-820f-e6401ea0255b] Successfully reverted task state from powering-on on failure for instance.
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server [req-ac9ab44c-c6bc-421c-8c6d-7a777cb4575e 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Exception during message handling: KeyError: 'pci_slot'
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/utils.py", line 1323, in decorated_function
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 2922, in start_instance
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/manager.py", line 2892, in _power_on
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     block_device_info)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2987, in power_on
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2855, in _hard_reboot
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     mdevs=mdevs)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5482, in _get_guest_xml
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     context, mdevs)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5282, in _get_guest_config
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     flavor, virt_type, self._host)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 608, in get_config
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     inst_type, virt_type, host)
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 353, in get_config_hw_veb
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server     conf, net_type, profile['pci_slot'],
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
  2021-01-18 17:19:04.657 1496 ERROR oslo_messaging.rpc.server

  ---------

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1912273/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next