yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1369508] [NEW] Instance with NUMA topology causes exception in the scheduler

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nikola Đipanov <ndipanov@xxxxxxxxxx>
Date: Mon, 15 Sep 2014 10:47:53 -0000
Reply-to: Bug 1369508 <1369508@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This was reported by Michael Turek as he was testing this while the
patches were still in flight See:
https://review.openstack.org/#/c/114938/26/nova/virt/hardware.py

As described on there - the code there makes a bad assumption about the
format in which it will get the data in the scheduler, which results in:

2014-09-15 10:45:44.906 ERROR oslo.messaging.rpc.dispatcher [req-f29a469e-268d-49bf-abfa-0ccb228d768c admin admin] Exception during message handling: An object of type InstanceNUMACell is required here
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/manager.py", line 175, in select_destinations
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 147, in select_destinations
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 300, in _schedule
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     chosen_host.obj.consume_from_instance(context, instance_properties)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 252, in consume_from_instance
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self, instance)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/hardware.py", line 978, in get_host_numa_usage_from_instance
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     instance_numa_topology = instance_topology_from_instance(instance)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/hardware.py", line 949, in instance_topology_from_instance
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     cells=cells)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 242, in __init__
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self[key] = kwargs[key]
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 474, in __setitem__
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     setattr(self, name, value)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 75, in setter
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     field_value = field.coerce(self, name, value)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 189, in coerce
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._type.coerce(obj, attr, value)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 388, in coerce
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     obj, '%s[%i]' % (attr, index), element)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 189, in coerce
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._type.coerce(obj, attr, value)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 474, in coerce
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self._obj_name)
2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher ValueError: An object of type InstanceNUMACell is required here

** Affects: nova
     Importance: High
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1369508

Title:
  Instance with NUMA topology causes exception in the scheduler

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  This was reported by Michael Turek as he was testing this while the
  patches were still in flight See:
  https://review.openstack.org/#/c/114938/26/nova/virt/hardware.py

  As described on there - the code there makes a bad assumption about
  the format in which it will get the data in the scheduler, which
  results in:

  2014-09-15 10:45:44.906 ERROR oslo.messaging.rpc.dispatcher [req-f29a469e-268d-49bf-abfa-0ccb228d768c admin admin] Exception during message handling: An object of type InstanceNUMACell is required here
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/manager.py", line 175, in select_destinations
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 147, in select_destinations
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 300, in _schedule
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     chosen_host.obj.consume_from_instance(context, instance_properties)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 252, in consume_from_instance
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self, instance)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/hardware.py", line 978, in get_host_numa_usage_from_instance
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     instance_numa_topology = instance_topology_from_instance(instance)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/hardware.py", line 949, in instance_topology_from_instance
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     cells=cells)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 242, in __init__
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self[key] = kwargs[key]
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 474, in __setitem__
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     setattr(self, name, value)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/base.py", line 75, in setter
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     field_value = field.coerce(self, name, value)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 189, in coerce
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._type.coerce(obj, attr, value)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 388, in coerce
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     obj, '%s[%i]' % (attr, index), element)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 189, in coerce
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     return self._type.coerce(obj, attr, value)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/fields.py", line 474, in coerce
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher     self._obj_name)
  2014-09-15 10:45:44.906 TRACE oslo.messaging.rpc.dispatcher ValueError: An object of type InstanceNUMACell is required here

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1369508/+subscriptions
Follow ups

[Bug 1369508] Re: Instance with NUMA topology causes exception in the scheduler
From: Thierry Carrez, 2014-10-01
[Bug 1369508] [NEW] Instance with NUMA topology causes exception in the scheduler
From: Nikola Đipanov, 2014-09-15
References

[Bug 1369508] [NEW] Instance with NUMA topology causes exception in the scheduler
From: Nikola Đipanov, 2014-09-15