yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1952941] [NEW] Migration fails with "NotImplementedError: Cannot load 'pcpuset' in the base class" when a pre Victoria instance with cpu pinning is migrated in Victoria

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1952941@xxxxxxxxxxxxxxxxxx>
Date: Wed, 01 Dec 2021 17:48:20 -0000
Reply-to: Bug 1952941 <1952941@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When the cpuset -> pcpuset data migration was added to
InstanceNUMATopology [1] it was missed that such object is not only
hydrated via InstanceNUMATopology.get_by_instance_uuid() but also
hydrated by RequestSpec.get_by_instance_uuid() indirectly. However the
latter code patch does not call InstanceNUMATopology.obj_from_db_obj()
that triggers the data migration via
InstanceNUMATopology._migrate_legacy_dedicated_instance_cpuset. This
causes that when the new nova code loads an old RequestSpec object from
the DB (e.g. during migration of an instance) the InstanceNUMATopology
in the RequestSpec will not be migrated to the new object version and it
will lead to errors when the pcpuset field is read during scheduling.

To reproduce:
* Install a pre Victoria cloud
* Create an instance with cpu pinning
* Upgrade to Victoria or newer
* Try to migrate / evacuate the instance

You will see the following stack trace in the nova-scheduler log

2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 210, in _schedule
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 441, in _get_sorted_hosts
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     spec_obj, index)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py", line 606, in get_filtered_hosts
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     hosts, spec_obj, index)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/filters.py", line 88, in get_filtered_objects
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     list_objs = list(objs)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/filters.py", line 43, in filter_all
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     if self._filter_one(obj, spec_obj):
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return self.host_passes(obj, spec)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 104, in host_passes
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     pci_stats=host_state.pci_stats))
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/hardware.py", line 2294, in numa_fit_instance_to_host
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     host_cell, instance_cell, limits, cpuset_reserved)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/hardware.py", line 1109, in _numa_fit_instance_cell
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     required_cpus = len(instance_cell.pcpuset) + cpuset_reserved
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     self.obj_load_attr(name)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 601, in obj_load_attr
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     _("Cannot load '%s' in the base class") % attrname)
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server NotImplementedError: Cannot load 'pcpuset' in the base class
2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server


[1] https://review.opendev.org/c/openstack/nova/+/714658

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: upgrade

** Summary changed:

- NotImplementedError: Cannot load 'pcpuset' in the base class if a pre Victoria instance with cpu pinning is migrated in Victoria
+ Migration fails with "NotImplementedError: Cannot load 'pcpuset' in the base class" when a pre Victoria instance with cpu pinning is migrated in Victoria

** Tags added: upgrade

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1952941

Title:
  Migration fails with "NotImplementedError: Cannot load 'pcpuset' in
  the base class" when a pre Victoria instance with cpu pinning is
  migrated in Victoria

Status in OpenStack Compute (nova):
  New

Bug description:
  When the cpuset -> pcpuset data migration was added to
  InstanceNUMATopology [1] it was missed that such object is not only
  hydrated via InstanceNUMATopology.get_by_instance_uuid() but also
  hydrated by RequestSpec.get_by_instance_uuid() indirectly. However the
  latter code patch does not call InstanceNUMATopology.obj_from_db_obj()
  that triggers the data migration via
  InstanceNUMATopology._migrate_legacy_dedicated_instance_cpuset. This
  causes that when the new nova code loads an old RequestSpec object
  from the DB (e.g. during migration of an instance) the
  InstanceNUMATopology in the RequestSpec will not be migrated to the
  new object version and it will lead to errors when the pcpuset field
  is read during scheduling.

  To reproduce:
  * Install a pre Victoria cloud
  * Create an instance with cpu pinning
  * Upgrade to Victoria or newer
  * Try to migrate / evacuate the instance

  You will see the following stack trace in the nova-scheduler log

  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 210, in _schedule
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 441, in _get_sorted_hosts
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     spec_obj, index)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py", line 606, in get_filtered_hosts
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     hosts, spec_obj, index)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/filters.py", line 88, in get_filtered_objects
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     list_objs = list(objs)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/filters.py", line 43, in filter_all
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     if self._filter_one(obj, spec_obj):
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     return self.host_passes(obj, spec)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 104, in host_passes
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     pci_stats=host_state.pci_stats))
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/hardware.py", line 2294, in numa_fit_instance_to_host
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     host_cell, instance_cell, limits, cpuset_reserved)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/hardware.py", line 1109, in _numa_fit_instance_cell
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     required_cpus = len(instance_cell.pcpuset) + cpuset_reserved
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     self.obj_load_attr(name)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 601, in obj_load_attr
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server     _("Cannot load '%s' in the base class") % attrname)
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server NotImplementedError: Cannot load 'pcpuset' in the base class
  2021-11-30 17:36:38.963 48 ERROR oslo_messaging.rpc.server

  
  [1] https://review.opendev.org/c/openstack/nova/+/714658

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1952941/+subscriptions
Follow ups

[Bug 1952941] Fix included in openstack/nova 24.1.0
From: OpenStack Infra, 2022-03-10
[Bug 1952941] Re: Migration fails with "NotImplementedError: Cannot load 'pcpuset' in the base class" when a pre Victoria instance with cpu pinning is migrated in Victoria
From: OpenStack Infra, 2022-02-04
[Bug 1952941] Re: Migration fails with "NotImplementedError: Cannot load 'pcpuset' in the base class" when a pre Victoria instance with cpu pinning is migrated in Victoria
From: Balazs Gibizer, 2021-12-02