← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1816454] [NEW] hw:mem_page_size is not respecting all documented values

 

Public bug reported:

Per the Rocky documentation for hugepages:
https://docs.openstack.org/nova/rocky/admin/huge-pages.html

2MB hugepages can be specified either as:
--property hw:mem_page_size=2Mb, or
--property hw:mem_page_size=2048

However, whenever I use the former notation (2Mb), conductor fails with
the misleading NUMA error below... whereas with the latter notation
(2048), allocation succeeds and the resulting instance is backed with
2MB hugepages on an x86_64 platform (as verified by checking
`/proc/meminfo | grep HugePages_Free` before/after stopping the created
instance).

ERROR nova.scheduler.utils [req-de6920d5-829b-411c-acd7-1343f48824c9
cb2abbb91da54209a5ad93a845b4cc26 cb226ff7932d40b0a48ec129e162a2fb -
default default] [instance: 5b53d1d4-6a16-4db9-ab52-b267551c6528] Error
from last host: node1 (node FQDN-REDACTED): ['Traceback (most recent
call last):\n', '  File "/usr/lib/python3/dist-
packages/nova/compute/manager.py", line 2106, in
_build_and_run_instance\n    with rt.instance_claim(context, instance,
node, limits):\n', '  File "/usr/lib/python3/dist-
packages/oslo_concurrency/lockutils.py", line 274, in inner\n    return
f(*args, **kwargs)\n', '  File "/usr/lib/python3/dist-
packages/nova/compute/resource_tracker.py", line 217, in
instance_claim\n    pci_requests, overhead=overhead, limits=limits)\n',
'  File "/usr/lib/python3/dist-packages/nova/compute/claims.py", line
95, in __init__\n    self._claim_test(resources, limits)\n', '  File
"/usr/lib/python3/dist-packages/nova/compute/claims.py", line 162, in
_claim_test\n    "; ".join(reasons))\n',
'nova.exception.ComputeResourcesUnavailable: Insufficient compute
resources: Requested instance NUMA topology cannot fit the given host
NUMA topology.\n', '\nDuring handling of the above exception, another
exception occurred:\n\n', 'Traceback (most recent call last):\n', '
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line
1940, in _do_build_and_run_instance\n    filter_properties,
request_spec)\n', '  File "/usr/lib/python3/dist-
packages/nova/compute/manager.py", line 2156, in
_build_and_run_instance\n    instance_uuid=instance.uuid,
reason=e.format_message())\n', 'nova.exception.RescheduledException:
Build of instance 5b53d1d4-6a16-4db9-ab52-b267551c6528 was re-scheduled:
Insufficient compute resources: Requested instance NUMA topology cannot
fit the given host NUMA topology.\n']

Additional info:
I am using Debian testing (buster) and all OpenStack packages included therein.

$ dpkg -l | grep nova
ii  nova-common                           2:18.1.0-2                              all          OpenStack Compute - common files
ii  nova-compute                          2:18.1.0-2                              all          OpenStack Compute - compute node
ii  nova-compute-kvm                      2:18.1.0-2                              all          OpenStack Compute - compute node (KVM)
ii  python3-nova                          2:18.1.0-2                              all          OpenStack Compute - libraries
ii  python3-novaclient                    2:11.0.0-2                              all          client library for OpenStack Compute API - 3.x

$ dpkg -l | grep qemu
ii  ipxe-qemu                             1.0.0+git-20161027.b991c67-1            all          PXE boot firmware - ROM images for qemu
ii  qemu-block-extra:amd64                1:3.1+dfsg-2+b1                         amd64        extra block backend modules for qemu-system and qemu-utils
ii  qemu-kvm                              1:3.1+dfsg-2+b1                         amd64        QEMU Full virtualization on x86 hardware
ii  qemu-system-common                    1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (common files)
ii  qemu-system-data                      1:3.1+dfsg-2                            all          QEMU full system emulation (data files)
ii  qemu-system-gui                       1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (user interface and audio support)
ii  qemu-system-x86                       1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (x86)
ii  qemu-utils                            1:3.1+dfsg-2+b1                         amd64        QEMU utilities

* I forced nova to allocate on the same hypervisor (node1) when checking
for the issue and can repeatedly allocate using a flavor which specifies
hugepages with hw:mem_page_size=2048 -- on the contrary, when using a
flavor which is otherwise unchanged except for the 2048/2Mb difference,
allocation repeatedly fails.

* I am using libvirt+kvm.  I don't think it matters, but I am using Ceph
as a storage backend and neutron in a very basic VLAN-based segmentation
configuration (no OVS or anything remotely fancy).

* I specified hw:numa_nodes='1' when creating the flavor... and all my
hypervisors only have 1 NUMA node, so allocation should always succeed
as long as there are free huge pages (which there are).

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1816454

Title:
  hw:mem_page_size is not respecting all documented values

Status in OpenStack Compute (nova):
  New

Bug description:
  Per the Rocky documentation for hugepages:
  https://docs.openstack.org/nova/rocky/admin/huge-pages.html

  2MB hugepages can be specified either as:
  --property hw:mem_page_size=2Mb, or
  --property hw:mem_page_size=2048

  However, whenever I use the former notation (2Mb), conductor fails
  with the misleading NUMA error below... whereas with the latter
  notation (2048), allocation succeeds and the resulting instance is
  backed with 2MB hugepages on an x86_64 platform (as verified by
  checking `/proc/meminfo | grep HugePages_Free` before/after stopping
  the created instance).

  ERROR nova.scheduler.utils [req-de6920d5-829b-411c-acd7-1343f48824c9
  cb2abbb91da54209a5ad93a845b4cc26 cb226ff7932d40b0a48ec129e162a2fb -
  default default] [instance: 5b53d1d4-6a16-4db9-ab52-b267551c6528]
  Error from last host: node1 (node FQDN-REDACTED): ['Traceback (most
  recent call last):\n', '  File "/usr/lib/python3/dist-
  packages/nova/compute/manager.py", line 2106, in
  _build_and_run_instance\n    with rt.instance_claim(context, instance,
  node, limits):\n', '  File "/usr/lib/python3/dist-
  packages/oslo_concurrency/lockutils.py", line 274, in inner\n
  return f(*args, **kwargs)\n', '  File "/usr/lib/python3/dist-
  packages/nova/compute/resource_tracker.py", line 217, in
  instance_claim\n    pci_requests, overhead=overhead,
  limits=limits)\n', '  File "/usr/lib/python3/dist-
  packages/nova/compute/claims.py", line 95, in __init__\n
  self._claim_test(resources, limits)\n', '  File "/usr/lib/python3
  /dist-packages/nova/compute/claims.py", line 162, in _claim_test\n
  "; ".join(reasons))\n', 'nova.exception.ComputeResourcesUnavailable:
  Insufficient compute resources: Requested instance NUMA topology
  cannot fit the given host NUMA topology.\n', '\nDuring handling of the
  above exception, another exception occurred:\n\n', 'Traceback (most
  recent call last):\n', '  File "/usr/lib/python3/dist-
  packages/nova/compute/manager.py", line 1940, in
  _do_build_and_run_instance\n    filter_properties, request_spec)\n', '
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line
  2156, in _build_and_run_instance\n    instance_uuid=instance.uuid,
  reason=e.format_message())\n', 'nova.exception.RescheduledException:
  Build of instance 5b53d1d4-6a16-4db9-ab52-b267551c6528 was re-
  scheduled: Insufficient compute resources: Requested instance NUMA
  topology cannot fit the given host NUMA topology.\n']

  Additional info:
  I am using Debian testing (buster) and all OpenStack packages included therein.

  $ dpkg -l | grep nova
  ii  nova-common                           2:18.1.0-2                              all          OpenStack Compute - common files
  ii  nova-compute                          2:18.1.0-2                              all          OpenStack Compute - compute node
  ii  nova-compute-kvm                      2:18.1.0-2                              all          OpenStack Compute - compute node (KVM)
  ii  python3-nova                          2:18.1.0-2                              all          OpenStack Compute - libraries
  ii  python3-novaclient                    2:11.0.0-2                              all          client library for OpenStack Compute API - 3.x

  $ dpkg -l | grep qemu
  ii  ipxe-qemu                             1.0.0+git-20161027.b991c67-1            all          PXE boot firmware - ROM images for qemu
  ii  qemu-block-extra:amd64                1:3.1+dfsg-2+b1                         amd64        extra block backend modules for qemu-system and qemu-utils
  ii  qemu-kvm                              1:3.1+dfsg-2+b1                         amd64        QEMU Full virtualization on x86 hardware
  ii  qemu-system-common                    1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (common files)
  ii  qemu-system-data                      1:3.1+dfsg-2                            all          QEMU full system emulation (data files)
  ii  qemu-system-gui                       1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (user interface and audio support)
  ii  qemu-system-x86                       1:3.1+dfsg-2+b1                         amd64        QEMU full system emulation binaries (x86)
  ii  qemu-utils                            1:3.1+dfsg-2+b1                         amd64        QEMU utilities

  * I forced nova to allocate on the same hypervisor (node1) when
  checking for the issue and can repeatedly allocate using a flavor
  which specifies hugepages with hw:mem_page_size=2048 -- on the
  contrary, when using a flavor which is otherwise unchanged except for
  the 2048/2Mb difference, allocation repeatedly fails.

  * I am using libvirt+kvm.  I don't think it matters, but I am using
  Ceph as a storage backend and neutron in a very basic VLAN-based
  segmentation configuration (no OVS or anything remotely fancy).

  * I specified hw:numa_nodes='1' when creating the flavor... and all my
  hypervisors only have 1 NUMA node, so allocation should always succeed
  as long as there are free huge pages (which there are).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1816454/+subscriptions


Follow ups