yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1513808] [NEW] cannot schedule instances in NUMA topology with more than 6 vcpus using SR-IOV ports

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pedro Sousa <1513808@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Nov 2015 11:39:09 -0000
Reply-to: Bug 1513808 <1513808@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi all,

I have a Centos 7.1 kilo deployment, using sr-iov ports to my instances.
I'm trying to configure NUMA topology and CPU pinning for some telco
based workloads.

I have 3 compute nodes, I'm trying to use one of them to use cpu
pinning.

I've configured it like this:
Compute Node (total 24 cpus)

/etc/nova/nova.conf
vcpu_pin_set=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23

Changed grub to isolate my cpus:
#grubby --update-kernel=ALL --args="isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23"
#grub2-install /dev/sda


Controller Nodes:

/etc/nova/nova.conf
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

Created host aggregate performance 
#nova aggregate-create performance
#nova aggregate-set-metadata 1 pinned=true
#nova aggregate-add-host 1 compute03

Created host aggregate normal
#nova aggregate-create normal
#nova aggregate-set-metadata 2 pinned=false
#nova aggregate-add-host 2 compute01
#nova aggregate-add-host 2 compute02

Created the flavor with cpu pinning
#nova flavor-create m1.performance 6 2048 20 4
#nova flavor-key 6 set hw:cpu_policy=dedicated
#nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true

The issue is:
With SR-IOV ports it only let's me create instances with 6 vcpus in total with the conf described above. Without SR-IOV, using OVS, I don't have that limitation. Is this a bug or something? I've seen this: https://bugs.launchpad.net/nova/+bug/1441169, however I have the patch, and as I said it works for the first 6 vcpus with my configuration. 

Some relevant logs:
/var/log/nova/nova-scheduler.log
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.957 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter PciPassthroughFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
2015-11-06 11:18:17.959 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter NUMATopologyFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1513808

Title:
  cannot schedule instances in NUMA topology with more than 6 vcpus
  using SR-IOV ports

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi all,

  I have a Centos 7.1 kilo deployment, using sr-iov ports to my
  instances. I'm trying to configure NUMA topology and CPU pinning for
  some telco based workloads.

  I have 3 compute nodes, I'm trying to use one of them to use cpu
  pinning.

  I've configured it like this:
  Compute Node (total 24 cpus)

  /etc/nova/nova.conf
  vcpu_pin_set=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23

  Changed grub to isolate my cpus:
  #grubby --update-kernel=ALL --args="isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23"
  #grub2-install /dev/sda

  
  Controller Nodes:

  /etc/nova/nova.conf
  scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
  scheduler_available_filters = nova.scheduler.filters.all_filters
  scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

  Created host aggregate performance 
  #nova aggregate-create performance
  #nova aggregate-set-metadata 1 pinned=true
  #nova aggregate-add-host 1 compute03

  Created host aggregate normal
  #nova aggregate-create normal
  #nova aggregate-set-metadata 2 pinned=false
  #nova aggregate-add-host 2 compute01
  #nova aggregate-add-host 2 compute02

  Created the flavor with cpu pinning
  #nova flavor-create m1.performance 6 2048 20 4
  #nova flavor-key 6 set hw:cpu_policy=dedicated
  #nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true

  The issue is:
  With SR-IOV ports it only let's me create instances with 6 vcpus in total with the conf described above. Without SR-IOV, using OVS, I don't have that limitation. Is this a bug or something? I've seen this: https://bugs.launchpad.net/nova/+bug/1441169, however I have the patch, and as I said it works for the first 6 vcpus with my configuration. 

  Some relevant logs:
  /var/log/nova/nova-scheduler.log
  2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70
  2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.957 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter PciPassthroughFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84
  2015-11-06 11:18:17.959 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter NUMATopologyFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1513808/+subscriptions
Follow ups

[Bug 1513808] Re: cannot schedule instances in NUMA topology with more than 6 vcpus using SR-IOV ports
From: Markus Zoeller (markus_z), 2016-07-13