yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1397381] [NEW] numa cell ids need to be normalized before creating xml

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@xxxxxxxxx>
Date: Fri, 28 Nov 2014 16:45:53 -0000
Reply-to: Bug 1397381 <1397381@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When creating instance with numa topology, if that instance is placed on
a host numa node different then 0 , invalid libvirt.xml will be
generated. it's because instance cell id's are used to store host id
assignment. Instance cell id's should be normalized before generating
libvirt.xml


2014-11-28 16:35:55.195 ERROR nova.virt.libvirt.driver [-] Error defining a domain with XML: <domain type="kvm">
  <uuid>b90dbaea-ed6f-49fa-b3bf-75137005c621</uuid>
  <name>instance-00000005</name>
  <memory>2097152</memory>
  <numatune>
    <memory mode="strict" nodeset="1"/>
    <memnode cellid="1" mode="strict" nodeset="1"/>
  </numatune>
  <vcpu>4</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="2015.1.0"/>
      <nova:name>vm1</nova:name>
      <nova:creationTime>2014-11-28 16:35:55</nova:creationTime>
      <nova:flavor name="nano_mini">
        <nova:memory>2048</nova:memory>
        <nova:disk>0</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="04d6fa121cbc4b60b57c96eb32b356b1">admin</nova:user>
        <nova:project uuid="9193517bf6c64d0296d60095c88b2e5a">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="c0fc1ed6-1ee5-47fa-a2e7-f1aee3c6750b"/>
    </nova:instance>
  </metadata>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2015.1.0</entry>
      <entry name="serial">10881668-12bf-49c2-9efb-28bb8d278929</entry>
      <entry name="uuid">b90dbaea-ed6f-49fa-b3bf-75137005c621</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cputune>
    <vcpupin vcpu="0" cpuset="10-19,30-39"/>
    <vcpupin vcpu="1" cpuset="10-19,30-39"/>
    <vcpupin vcpu="2" cpuset="10-19,30-39"/>
    <vcpupin vcpu="3" cpuset="10-19,30-39"/>
  </cputune>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="hpet" present="no"/>
  </clock>
  <cpu match="exact">
    <topology sockets="4" cores="1" threads="1"/>
    <numa>
      <cell id="1" cpus="0-3" memory="2097152"/>
    </numa>
  </cpu>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="none"/>
      <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk.config"/>
      <target bus="ide" dev="hdd"/>
    </disk>
    <interface type="hostdev" managed="yes">
      <mac address="fa:16:3e:69:8c:17"/>
      <source>
        <address type="pci" domain="0x0000" bus="0x82" slot="0x10" function="0x2"/>
      </source>
      <vlan>
        <tag id="1000"/>
      </vlan>
    </interface>
    <serial type="file">
      <source path="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/console.log"/>
    </serial>
    <serial type="pty"/>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
    <video>
      <model type="cirrus"/>
    </video>
    <memballoon model="virtio">
      <stats period="10"/>
    </memballoon>
  </devices>
</domain>

2014-11-28 16:35:55.195 ERROR nova.compute.manager [-] [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Instance failed to spawn
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Traceback (most recent call last):
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/compute/manager.py", line 2246, in _build_resources
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     yield resources
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/compute/manager.py", line 2116, in _build_and_run_instance
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     instance_type=instance_type)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 2641, in spawn
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     block_device_info, disk_info=disk_info)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4490, in _create_domain_and_network
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     power_on=power_on)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     LOG.error(err)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     six.reraise(self.type_, self.value, self.tb)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4407, in _create_domain
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     domain = self._conn.defineXML(xml)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     rv = execute(f, *args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     six.reraise(c, e, tb)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     rv = meth(*args, **kwargs)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3484, in defineXML
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] libvirtError: XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed

** Affects: nova
     Importance: Undecided
     Assignee: Przemyslaw Czesnowicz (pczesno)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Przemyslaw Czesnowicz (pczesno)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1397381

Title:
  numa cell ids need to be normalized before creating  xml

Status in OpenStack Compute (Nova):
  New

Bug description:
  When creating instance with numa topology, if that instance is placed
  on a host numa node different then 0 , invalid libvirt.xml will be
  generated. it's because instance cell id's are used to store host id
  assignment. Instance cell id's should be normalized before generating
  libvirt.xml

  
  2014-11-28 16:35:55.195 ERROR nova.virt.libvirt.driver [-] Error defining a domain with XML: <domain type="kvm">
    <uuid>b90dbaea-ed6f-49fa-b3bf-75137005c621</uuid>
    <name>instance-00000005</name>
    <memory>2097152</memory>
    <numatune>
      <memory mode="strict" nodeset="1"/>
      <memnode cellid="1" mode="strict" nodeset="1"/>
    </numatune>
    <vcpu>4</vcpu>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="2015.1.0"/>
        <nova:name>vm1</nova:name>
        <nova:creationTime>2014-11-28 16:35:55</nova:creationTime>
        <nova:flavor name="nano_mini">
          <nova:memory>2048</nova:memory>
          <nova:disk>0</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>4</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="04d6fa121cbc4b60b57c96eb32b356b1">admin</nova:user>
          <nova:project uuid="9193517bf6c64d0296d60095c88b2e5a">demo</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="c0fc1ed6-1ee5-47fa-a2e7-f1aee3c6750b"/>
      </nova:instance>
    </metadata>
    <sysinfo type="smbios">
      <system>
        <entry name="manufacturer">OpenStack Foundation</entry>
        <entry name="product">OpenStack Nova</entry>
        <entry name="version">2015.1.0</entry>
        <entry name="serial">10881668-12bf-49c2-9efb-28bb8d278929</entry>
        <entry name="uuid">b90dbaea-ed6f-49fa-b3bf-75137005c621</entry>
      </system>
    </sysinfo>
    <os>
      <type>hvm</type>
      <boot dev="hd"/>
      <smbios mode="sysinfo"/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cputune>
      <vcpupin vcpu="0" cpuset="10-19,30-39"/>
      <vcpupin vcpu="1" cpuset="10-19,30-39"/>
      <vcpupin vcpu="2" cpuset="10-19,30-39"/>
      <vcpupin vcpu="3" cpuset="10-19,30-39"/>
    </cputune>
    <clock offset="utc">
      <timer name="pit" tickpolicy="delay"/>
      <timer name="rtc" tickpolicy="catchup"/>
      <timer name="hpet" present="no"/>
    </clock>
    <cpu match="exact">
      <topology sockets="4" cores="1" threads="1"/>
      <numa>
        <cell id="1" cpus="0-3" memory="2097152"/>
      </numa>
    </cpu>
    <devices>
      <disk type="file" device="disk">
        <driver name="qemu" type="qcow2" cache="none"/>
        <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk"/>
        <target bus="virtio" dev="vda"/>
      </disk>
      <disk type="file" device="cdrom">
        <driver name="qemu" type="raw" cache="none"/>
        <source file="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/disk.config"/>
        <target bus="ide" dev="hdd"/>
      </disk>
      <interface type="hostdev" managed="yes">
        <mac address="fa:16:3e:69:8c:17"/>
        <source>
          <address type="pci" domain="0x0000" bus="0x82" slot="0x10" function="0x2"/>
        </source>
        <vlan>
          <tag id="1000"/>
        </vlan>
      </interface>
      <serial type="file">
        <source path="/opt/stack/data/nova/instances/b90dbaea-ed6f-49fa-b3bf-75137005c621/console.log"/>
      </serial>
      <serial type="pty"/>
      <graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
      <video>
        <model type="cirrus"/>
      </video>
      <memballoon model="virtio">
        <stats period="10"/>
      </memballoon>
    </devices>
  </domain>

  2014-11-28 16:35:55.195 ERROR nova.compute.manager [-] [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Instance failed to spawn
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] Traceback (most recent call last):
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/compute/manager.py", line 2246, in _build_resources
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     yield resources
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/compute/manager.py", line 2116, in _build_and_run_instance
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     instance_type=instance_type)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 2641, in spawn
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     block_device_info, disk_info=disk_info)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4490, in _create_domain_and_network
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     power_on=power_on)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     LOG.error(err)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     six.reraise(self.type_, self.value, self.tb)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/shared/stack/nova/nova/virt/libvirt/driver.py", line 4407, in _create_domain
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     domain = self._conn.defineXML(xml)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     rv = execute(f, *args, **kwargs)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     six.reraise(c, e, tb)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     rv = meth(*args, **kwargs)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3484, in defineXML
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621]     if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
  2014-11-28 16:35:55.195 TRACE nova.compute.manager [instance: b90dbaea-ed6f-49fa-b3bf-75137005c621] libvirtError: XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1397381/+subscriptions
Follow ups

[Bug 1397381] Re: numa cell ids need to be normalized before creating xml
From: Thierry Carrez, 2014-12-18
[Bug 1397381] [NEW] numa cell ids need to be normalized before creating xml
From: Przemyslaw Czesnowicz, 2014-11-28
References

[Bug 1397381] [NEW] numa cell ids need to be normalized before creating xml
From: Przemyslaw Czesnowicz, 2014-11-28