yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1417201] Re: nova-scheduler exception when trying to use hugepages

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Fri, 20 Mar 2015 07:37:28 -0000
Reply-to: Bug 1417201 <1417201@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1417201

Title:
  nova-scheduler exception when trying to use hugepages

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I'm trying to make use of huge pages as described in
  "http://specs.openstack.org/openstack/nova-
  specs/specs/kilo/implemented/virt-driver-large-pages.html".  I'm
  running nova kilo as of Jan 27th.  The other openstack services are
  juno.  Libvirt is 1.2.8.

  I've allocated 10000 2MB pages on a compute node.  "virsh
  capabilities" on that node contains:

      <topology>
        <cells num='2'>
          <cell id='0'>
            <memory unit='KiB'>67028244</memory>
            <pages unit='KiB' size='4'>16032069</pages>
            <pages unit='KiB' size='2048'>5000</pages>
            <pages unit='KiB' size='1048576'>1</pages>
  ...
          <cell id='1'>
            <memory unit='KiB'>67108864</memory>
            <pages unit='KiB' size='4'>16052224</pages>
            <pages unit='KiB' size='2048'>5000</pages>
            <pages unit='KiB' size='1048576'>1</pages>

  I then restarted nova-compute, I set "hw:mem_page_size=large" on a
  flavor, and then tried to boot up an instance with that flavor.  I got
  the error logs below in nova-scheduler.  Is this a bug?

  Feb  2 16:23:10 controller-0 nova-scheduler Exception during message handling: Cannot load 'mempages' in the base class
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     filter_properties)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 138, in _schedule
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     filter_properties, index=num)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/host_manager.py", line 391, in get_filtered_hosts
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     hosts, filter_properties, index)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/filters.py", line 77, in get_filtered_objects
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     list_objs = list(objs)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/filters.py", line 43, in filter_all
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     if self._filter_one(obj, filter_properties):
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 27, in _filter_one
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     return self.host_passes(obj, filter_properties)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 45, in host_passes
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     limits_topology=limits))
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/virt/hardware.py", line 1161, in numa_fit_instance_to_host
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     host_cell, instance_cell, limit_cell)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/virt/hardware.py", line 851, in _numa_fit_instance_cell
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     host_cell, instance_cell)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/virt/hardware.py", line 692, in _numa_cell_supports_pagesize_request
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     avail_pagesize = [page.size_kb for page in host_cell.mempages]
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     self.obj_load_attr(name)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/nova/objects/base.py", line 507, in obj_load_attr
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher     _("Cannot load '%s' in the base class") % attrname)
  2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher NotImplementedError: Cannot load 'mempages' in the base class

  As far as nova-compute, at the end of
  nova.virt.libvirt.driver.LibvirtDriver.get_available_resource() I've
  confirmed that data['numa_topology'] looks like this:

  '{"nova_object.version": "1.2", "nova_object.changes": ["cells"],
  "nova_object.name": "NUMATopology", "nova_object.data": {"cells":
  [{"nova_object.version": "1.2", "nova_object.changes": ["cpu_usage",
  "memory_usage", "cpuset", "pinned_cpus", "siblings", "memory",
  "mempages", "id"], "nova_object.name": "NUMACell", "nova_object.data":
  {"cpu_usage": 0, "memory_usage": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7,
  8, 9, 10, 11], "pinned_cpus": [], "siblings": [], "memory": 65457,
  "mempages": [{"nova_object.version": "1.0", "nova_object.changes":
  ["total", "size_kb", "used"], "nova_object.name": "NUMAPagesTopology",
  "nova_object.data": {"total": 16032069, "used": 0, "size_kb": 4},
  "nova_object.namespace": "nova"}, {"nova_object.version": "1.0",
  "nova_object.changes": ["total", "size_kb", "used"],
  "nova_object.name": "NUMAPagesTopology", "nova_object.data": {"total":
  5000, "used": 0, "size_kb": 2048}, "nova_object.namespace": "nova"},
  {"nova_object.version": "1.0", "nova_object.changes": ["total",
  "size_kb", "used"], "nova_object.name": "NUMAPagesTopology",
  "nova_object.data": {"total": 1, "used": 0, "size_kb": 1048576},
  "nova_object.namespace": "nova"}], "id": 0}, "nova_object.namespace":
  "nova"}, {"nova_object.version": "1.2", "nova_object.changes":
  ["cpu_usage", "memory_usage", "cpuset", "pinned_cpus", "siblings",
  "memory", "mempages", "id"], "nova_object.name": "NUMACell",
  "nova_object.data": {"cpu_usage": 0, "memory_usage": 0, "cpuset": [12,
  13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], "pinned_cpus": [],
  "siblings": [], "memory": 65536, "mempages": [{"nova_object.version":
  "1.0", "nova_object.changes": ["total", "size_kb", "used"],
  "nova_object.name": "NUMAPagesTopology", "nova_object.data": {"total":
  16052224, "used": 0, "size_kb": 4}, "nova_object.namespace": "nova"},
  {"nova_object.version": "1.0", "nova_object.changes": ["total",
  "size_kb", "used"], "nova_object.name": "NUMAPagesTopology",
  "nova_object.data": {"total": 5000, "used": 0, "size_kb": 2048},
  "nova_object.namespace": "nova"}, {"nova_object.version": "1.0",
  "nova_object.changes": ["total", "size_kb", "used"],
  "nova_object.name": "NUMAPagesTopology", "nova_object.data": {"total":
  1, "used": 0, "size_kb": 1048576}, "nova_object.namespace": "nova"}],
  "id": 1}, "nova_object.namespace": "nova"}]}, "nova_object.namespace":
  "nova"}'

  I printed out str(host_topology) in NUMATopologyFilter.host_passes()
  and it gave:

  Feb  2 17:07:43 controller-0 nova-scheduler host_topology:
  NUMATopology(cells=[NUMACell(UNKNOWN),NUMACell(1)])

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1417201/+subscriptions
References

[Bug 1417201] [NEW] nova-scheduler exception when trying to use hugepages
From: Chris Friesen, 2015-02-02