yahoo-eng-team team mailing list archive

[Stephen Finucane <stephen.finucane@xxxxxxxxx>]

Public bug reported:

#1438226 reported that CPU pinning was broken in select versions of
libvirt. Further investigation has highlighted issues with NUMA tuning
in general on these versions. The same error messages seen with when
configuring CPU pinning are seen when configuring NUMA tuning. The
results from testing, mostly duplicated from the aforementioned bug
report, are given below. Note that v1.2.10 is still being tested at this
time.

This is somewhat related to #1422775 ("nova libvirt driver assumes qemu
support for NUMA pinning").

---

# Testing Configuration

Testing was conducted in a container which provided a single-node,
Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with
devstack). The yum-provided libvirt and its dependencies were removed
and libvirt and libvirt-python were built and installed from source.

# Results

The results are as follows (currently incomplete):

    versions  status
    --------  ------
    1.2.9     ok
    1.2.9.1   ok
    1.2.9.2   fail
    1.2.10    ???
    1.2.11    ok
    1.2.12    ok

v1.2.9.2 is broken by this (backported) patch:

    https://www.redhat.com/archives/libvir-
list/2014-November/msg00275.html

This can be seen as commit

    e226772 (qemu: fix domain startup failing with 'strict' mode in
numatune)

# Error logs

v1.2.9.2 produces the following exception:

    Traceback (most recent call last):
      File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
        yield resources
      File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
        flavor=flavor)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
        block_device_info=block_device_info)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
        power_on=power_on)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
        LOG.error(err)
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
        six.reraise(self.type_, self.value, self.tb)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
        domain.createWithFlags(launch_flags)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
        result = proxy_call(self._autowrap, f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
        rv = execute(f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
        six.reraise(c, e, tb)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
        rv = meth(*args, **kwargs)
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
        if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    libvirtError: Failed to create controller cpu for group: No such file or directory

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1449028

Title:
  NUMA tuning broken in select libvirt versions

Status in OpenStack Compute (Nova):
  New

Bug description:
  #1438226 reported that CPU pinning was broken in select versions of
  libvirt. Further investigation has highlighted issues with NUMA tuning
  in general on these versions. The same error messages seen with when
  configuring CPU pinning are seen when configuring NUMA tuning. The
  results from testing, mostly duplicated from the aforementioned bug
  report, are given below. Note that v1.2.10 is still being tested at
  this time.

  This is somewhat related to #1422775 ("nova libvirt driver assumes
  qemu support for NUMA pinning").

  ---

  # Testing Configuration

  Testing was conducted in a container which provided a single-node,
  Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built
  with devstack). The yum-provided libvirt and its dependencies were
  removed and libvirt and libvirt-python were built and installed from
  source.

  # Results

  The results are as follows (currently incomplete):

      versions  status
      --------  ------
      1.2.9     ok
      1.2.9.1   ok
      1.2.9.2   fail
      1.2.10    ???
      1.2.11    ok
      1.2.12    ok

  v1.2.9.2 is broken by this (backported) patch:

      https://www.redhat.com/archives/libvir-
  list/2014-November/msg00275.html

  This can be seen as commit

      e226772 (qemu: fix domain startup failing with 'strict' mode in
  numatune)

  # Error logs

  v1.2.9.2 produces the following exception:

      Traceback (most recent call last):
        File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
          yield resources
        File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
          flavor=flavor)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
          block_device_info=block_device_info)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
          power_on=power_on)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
          LOG.error(err)
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
          six.reraise(self.type_, self.value, self.tb)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
          domain.createWithFlags(launch_flags)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
          result = proxy_call(self._autowrap, f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
          rv = execute(f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
          six.reraise(c, e, tb)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
          rv = meth(*args, **kwargs)
        File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
          if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      libvirtError: Failed to create controller cpu for group: No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1449028/+subscriptions