← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #30651

[Bug 1438226] [NEW] nova libvirt driver assumes libvirt support for CPU pinning

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

CPU pinning support was implemented as part of this blueprint:

    http://specs.openstack.org/openstack/nova-specs/specs/juno/approved
/virt-driver-cpu-pinning.html

However, CPU pinning support is broken in some libvirt versions
(summarized below), resulting in exceptions when attempting to schedule
instances with the 'hw:cpu_policy' flavor key.

We should add a libvirt version test against known broken versions and
use that to determine whether or not to support the flavor keys.

This is somewhat related to #1422775 ("nova libvirt driver assumes qemu
support for NUMA pinning").

---

# Testing Configuration

Testing was conducted in a container which provided a single-node,
Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with
devstack). The yum-provided libvirt and its dependencies were removed
and libvirt and libvirt-python were built and installed from source.

# Results

The results are as follows:

    versions  status
    --------  ------
    1.2.9     ok
    1.2.9.1   ok
    1.2.9.2   fail
    1.2.10    fail
    1.2.11    ok
    1.2.12    ok

v1.2.9.2 is broken by this (backported) patch:

    https://www.redhat.com/archives/libvir-
list/2014-November/msg00275.html

This can be seen as commit

    e226772 (qemu: fix domain startup failing with 'strict' mode in
numatune)

v1.2.10 inherits is broken at checkout but can be fixed by applying
these three patches (yes, one of these broke v1.2.9.2 - the irony is not
lost on me):

    [0/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00274.html
     - [1/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00273.html
     - [2/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00276.html
     - [3/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html

# Error logs

v1.2.9.2 produces the following exception:

    Traceback (most recent call last):
      File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
        yield resources
      File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
        flavor=flavor)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
        block_device_info=block_device_info)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
        power_on=power_on)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
        LOG.error(err)
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
        six.reraise(self.type_, self.value, self.tb)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
        domain.createWithFlags(launch_flags)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
        result = proxy_call(self._autowrap, f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
        rv = execute(f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
        six.reraise(c, e, tb)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
        rv = meth(*args, **kwargs)
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
        if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    libvirtError: Failed to create controller cpu for group: No such file or directory

v1.2.10 produces the following exception:

    Traceback (most recent call last):
      File "/opt/stack/nova/nova/compute/manager.py", line 2342, in _build_resources
        yield resources
      File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance
        block_device_info=block_device_info)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2356, in spawn
        block_device_info=block_device_info)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4375, in _create_domain_and_network
        power_on=power_on)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4306, in _create_domain
        LOG.error(err)
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
        six.reraise(self.type_, self.value, self.tb)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4296, in _create_domain
        domain.createWithFlags(launch_flags)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
        result = proxy_call(self._autowrap, f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
        rv = execute(f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
        six.reraise(c, e, tb)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
        rv = meth(*args, **kwargs)
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1037, in createWithFlags
        if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/system.slice/docker.service/machine.slice/machine-qemu\x2dinstance\x2d0000000a.scope/cpuset.mems': Device or resource busy

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1438226

Title:
  nova libvirt driver assumes libvirt support for CPU pinning

Status in OpenStack Compute (Nova):
  New

Bug description:
  CPU pinning support was implemented as part of this blueprint:

      http://specs.openstack.org/openstack/nova-
  specs/specs/juno/approved/virt-driver-cpu-pinning.html

  However, CPU pinning support is broken in some libvirt versions
  (summarized below), resulting in exceptions when attempting to
  schedule instances with the 'hw:cpu_policy' flavor key.

  We should add a libvirt version test against known broken versions and
  use that to determine whether or not to support the flavor keys.

  This is somewhat related to #1422775 ("nova libvirt driver assumes
  qemu support for NUMA pinning").

  ---

  # Testing Configuration

  Testing was conducted in a container which provided a single-node,
  Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built
  with devstack). The yum-provided libvirt and its dependencies were
  removed and libvirt and libvirt-python were built and installed from
  source.

  # Results

  The results are as follows:

      versions  status
      --------  ------
      1.2.9     ok
      1.2.9.1   ok
      1.2.9.2   fail
      1.2.10    fail
      1.2.11    ok
      1.2.12    ok

  v1.2.9.2 is broken by this (backported) patch:

      https://www.redhat.com/archives/libvir-
  list/2014-November/msg00275.html

  This can be seen as commit

      e226772 (qemu: fix domain startup failing with 'strict' mode in
  numatune)

  v1.2.10 inherits is broken at checkout but can be fixed by applying
  these three patches (yes, one of these broke v1.2.9.2 - the irony is
  not lost on me):

      [0/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00274.html
       - [1/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00273.html
       - [2/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00276.html
       - [3/3] https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html

  # Error logs

  v1.2.9.2 produces the following exception:

      Traceback (most recent call last):
        File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
          yield resources
        File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
          flavor=flavor)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
          block_device_info=block_device_info)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
          power_on=power_on)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
          LOG.error(err)
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
          six.reraise(self.type_, self.value, self.tb)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
          domain.createWithFlags(launch_flags)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
          result = proxy_call(self._autowrap, f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
          rv = execute(f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
          six.reraise(c, e, tb)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
          rv = meth(*args, **kwargs)
        File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
          if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      libvirtError: Failed to create controller cpu for group: No such file or directory

  v1.2.10 produces the following exception:

      Traceback (most recent call last):
        File "/opt/stack/nova/nova/compute/manager.py", line 2342, in _build_resources
          yield resources
        File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance
          block_device_info=block_device_info)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2356, in spawn
          block_device_info=block_device_info)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4375, in _create_domain_and_network
          power_on=power_on)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4306, in _create_domain
          LOG.error(err)
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
          six.reraise(self.type_, self.value, self.tb)
        File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4296, in _create_domain
          domain.createWithFlags(launch_flags)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
          result = proxy_call(self._autowrap, f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
          rv = execute(f, *args, **kwargs)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
          six.reraise(c, e, tb)
        File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
          rv = meth(*args, **kwargs)
        File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1037, in createWithFlags
          if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
      libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/system.slice/docker.service/machine.slice/machine-qemu\x2dinstance\x2d0000000a.scope/cpuset.mems': Device or resource busy

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1438226/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next