yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1805767] Re: The new numa topology in the new flavor extra specs weren't parsed when resize

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Elod Illes <1805767@xxxxxxxxxxxxxxxxxx>
Date: Tue, 24 Nov 2020 14:16:44 -0000
Reply-to: Bug 1805767 <1805767@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/train
   Importance: Undecided
       Status: New

** Changed in: nova/train
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1805767

Title:
  The new numa topology in the new flavor extra specs weren't parsed
  when resize

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) train series:
  Fix Released

Bug description:
  Env:
  host with 2 numa nodes.

  flavor n2 request two instance numa nodes
  flavor n3 request three instance numa nodes

  Reproduce:
  Boot an instance with flavor n2, which scheduled to the host.
  Resize the instance with n3.

  
  The scheduler logs:
  Nov 28 18:27:16 jfz1r03h15 nova-scheduler[47260]: DEBUG nova.virt.hardware [None req-953d07bf-8ead-4f21-bd64-1ab12244eec1 admin admin] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy=None,cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([0]),cpuset_reserved=None,id=0,memory=256,pagesize=None) on host_cell NUMACell(cpu_usage=1,cpuset=set([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53]),id=0,memory=128835,memory_usage=256,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pinned_cpus=set([]),siblings=[set([43,7]),set([16,52]),set([2,38]),set([8,44]),set([50,14]),set([0,36]),set([51,15]),set([1,37]),set([10,46]),set([11,47]),set([42,6]),set([41,5]),set([9,45]),set([3,39]),set([48,12]),set([49,13]),set([17,53]),set([40,4])]) {{(pid=48606) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1019}}

  
  Nov 28 18:27:16 jfz1r03h15 nova-scheduler[47260]: DEBUG nova.virt.hardware [None req-953d07bf-8ead-4f21-bd64-1ab12244eec1 admin admin] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy=None,cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([1]),cpuset_reserved=None,id=1,memory=256,pagesize=None) on host_cell NUMACell(cpu_usage=1,cpuset=set([18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71]),id=1,memory=129009,memory_usage=256,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pinned_cpus=set([]),siblings=[set([59,23]),set([65,29]),set([18,54]),set([34,70]),set([24,60]),set([33,69]),set([58,22]),set([67,31]),set([66,30]),set([26,62]),set([35,71]),set([57,21]),set([25,61]),set([19,55]),set([64,28]),set([32,68]),set([27,63]),set([56,20])]) {{(pid=48606) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1019}}

  
  As above, the scheduler only see two instance numa nodes. It means the new flavor extra specs weren't parsed.

  The nova-compute log:
  Nov 28 18:27:27 jfz1r03h15 nova-scheduler[47260]: DEBUG oslo_service.periodic_task [None req-7aff4535-fe99-48b4-bab9-d206d35412ff None None] Running periodic task SchedulerManager._run_periodic_tasks {{(pid=48606) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:219}}
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 187, in decorated_function
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/utils.py", line 1157, in decorated_function
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4234, in prep_resize
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     filter_properties, host_list)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4298, in _reschedule_resize_or_reraise
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     six.reraise(*exc_info)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4224, in prep_resize
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     node, migration, clean_shutdown)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4184, in _prep_resize
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     limits=limits) as claim:
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 258, in resize_claim
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     limits=limits)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 327, in _move_claim
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     limits=limits)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/claims.py", line 275, in __init__
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     overhead=overhead, limits=limits)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/claims.py", line 95, in __init__
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     self._claim_test(resources, limits)
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/claims.py", line 162, in _claim_test
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server     "; ".join(reasons))
  Nov 28 18:27:18 jfz1r03h15 nova-compute[52929]: ERROR oslo_messaging.rpc.server ComputeResourcesUnavailable: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.

  The host only has two numa nodes, but the request resizing to three
  instance numa node still schedule to the host. And the move claim
  refused the request.

  Expect:
  The request of instance numa nodes should be updated with new flavor. And the scheduling should make a right scheduling in the beginning.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1805767/+subscriptions
References

[Bug 1805767] [NEW] The new flavor extra specs weren't parsed when resize
From: Alex Xu, 2018-11-29