← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #69006

[Bug 1729784] [NEW] nova-compute.log display incorrect information for CPUPinningUnknown.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
When nova-compute occured CPUPinningUnknown exception, the information of nova-compute.log is wrong.

Steps to reproduce
==================
* I did create 6 instances with numa topology.
  [root@ctrl3 ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000002              running
 2     instance-00000006              running
 3     instance-0000000a              running
 4     instance-00000008              running
 5     instance-0000000c              running
 6     instance-00000004              running

[root@ctrl3 ~]# virsh vcpupin 1
VCPU: CPU Affinity
----------------------------------
   0: 28

[root@ctrl3 ~]# virsh vcpupin 2
VCPU: CPU Affinity
----------------------------------
   0: 2

[root@ctrl3 ~]# virsh vcpupin 3
VCPU: CPU Affinity
----------------------------------
   0: 44

[root@ctrl3 ~]# virsh vcpupin 4
VCPU: CPU Affinity
----------------------------------
   0: 0

[root@ctrl3 ~]# virsh vcpupin 5
VCPU: CPU Affinity
----------------------------------
   0: 16

[root@ctrl3 ~]# virsh vcpupin 6
VCPU: CPU Affinity
----------------------------------
   0: 18

* then I did modify section vcpu_pin_set in nova.conf.
   [root@ctrl3 ~]# cat /etc/nova/nova.conf | grep vcpu_pin_set
    vcpu_pin_set = "0-47,^18"

* then I did restart openstack-nova-compute service
   Log display "Error updating resources for node ctrl3.: CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]".
   But this information is wrong, known CPU set is not only [28], it should be [0, 2, 4, 6, 8, 10, 12, 14, 16, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46]

Expected result
===============
CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [0, 2, 4, 6, 8, 10, 12, 14, 16, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46]

Actual result
=============
 CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]

Environment
===========
1. Exact version of OpenStack you are running. See the following
  Pike version

2. Which hypervisor did you use?
  Libvirt + KVM

Logs & Configs
==============
/lockutils.py:282
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager [req-8835b3d7-a453-4f91-abfa-5ccd0c54693b - - - - -] Error updating resources for node ctrl3.: CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager Traceback (most recent call last):
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6647, in update_available_resource_for_node
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 657, in update_available_resource
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     return f(*args, **kwargs)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 698, in _update_available_resource
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     self._update_usage_from_instances(context, instances, nodename)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1141, in _update_usage_from_instances
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     has_ocata_computes=has_ocata_computes)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1077, in _update_usage_from_instance
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     sign=sign)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 875, in _update_usage
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     cn, usage, free)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1784, in get_host_numa_usage_from_instance
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     host_numa_topology, instance_numa_topology, free=free))
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1640, in numa_usage_from_instances
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     newcell.pin_cpus(pinned_cpus)
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/objects/numa.py", line 82, in pin_cpus
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     cpuset=list(self.pinned_cpus))
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]
2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager

** Affects: nova
     Importance: Low
     Assignee: Charlotte Han (hanrong)
         Status: New

** Changed in: nova
   Importance: Undecided => Low

** Changed in: nova
     Assignee: (unassigned) => Charlotte Han (hanrong)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1729784

Title:
  nova-compute.log display incorrect information for CPUPinningUnknown.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When nova-compute occured CPUPinningUnknown exception, the information of nova-compute.log is wrong.

  Steps to reproduce
  ==================
  * I did create 6 instances with numa topology.
    [root@ctrl3 ~]# virsh list
   Id    Name                           State
  ----------------------------------------------------
   1     instance-00000002              running
   2     instance-00000006              running
   3     instance-0000000a              running
   4     instance-00000008              running
   5     instance-0000000c              running
   6     instance-00000004              running

  [root@ctrl3 ~]# virsh vcpupin 1
  VCPU: CPU Affinity
  ----------------------------------
     0: 28

  [root@ctrl3 ~]# virsh vcpupin 2
  VCPU: CPU Affinity
  ----------------------------------
     0: 2

  [root@ctrl3 ~]# virsh vcpupin 3
  VCPU: CPU Affinity
  ----------------------------------
     0: 44

  [root@ctrl3 ~]# virsh vcpupin 4
  VCPU: CPU Affinity
  ----------------------------------
     0: 0

  [root@ctrl3 ~]# virsh vcpupin 5
  VCPU: CPU Affinity
  ----------------------------------
     0: 16

  [root@ctrl3 ~]# virsh vcpupin 6
  VCPU: CPU Affinity
  ----------------------------------
     0: 18

  * then I did modify section vcpu_pin_set in nova.conf.
     [root@ctrl3 ~]# cat /etc/nova/nova.conf | grep vcpu_pin_set
      vcpu_pin_set = "0-47,^18"

  * then I did restart openstack-nova-compute service
     Log display "Error updating resources for node ctrl3.: CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]".
     But this information is wrong, known CPU set is not only [28], it should be [0, 2, 4, 6, 8, 10, 12, 14, 16, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46]

  Expected result
  ===============
  CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [0, 2, 4, 6, 8, 10, 12, 14, 16, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46]

  Actual result
  =============
   CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    Pike version

  2. Which hypervisor did you use?
    Libvirt + KVM

  Logs & Configs
  ==============
  /lockutils.py:282
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager [req-8835b3d7-a453-4f91-abfa-5ccd0c54693b - - - - -] Error updating resources for node ctrl3.: CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager Traceback (most recent call last):
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6647, in update_available_resource_for_node
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 657, in update_available_resource
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     return f(*args, **kwargs)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 698, in _update_available_resource
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     self._update_usage_from_instances(context, instances, nodename)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1141, in _update_usage_from_instances
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     has_ocata_computes=has_ocata_computes)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1077, in _update_usage_from_instance
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     sign=sign)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 875, in _update_usage
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     cn, usage, free)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1784, in get_host_numa_usage_from_instance
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     host_numa_topology, instance_numa_topology, free=free))
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1640, in numa_usage_from_instances
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     newcell.pin_cpus(pinned_cpus)
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/objects/numa.py", line 82, in pin_cpus
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager     cpuset=list(self.pinned_cpus))
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager CPUPinningUnknown: CPU set to pin [18] must be a subset of known CPU set [28]
  2017-11-03 14:44:01.425 32568 ERROR nova.compute.manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1729784/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next