yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1533572] Re: Failed to create vm on kvm while boot with multiple nics

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1533572@xxxxxxxxxxxxxxxxxx>
Date: Tue, 15 Mar 2016 04:17:31 -0000
Reply-to: Bug 1533572 <1533572@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1533572

Title:
  Failed to create vm on kvm while boot with multiple nics

Status in OpenStack Compute (nova):
  Expired

Bug description:
  On devsatck master branch, tried to boot vm with 4 vnic and it failed
  with error:- Unable to create tap device tap791b4be3-3c: Device or
  resource busy, 2 out of 20 times

  
  ERROR trace:-
  2016-01-13 21:41:04.550 ^[[01;31mERROR nova.compute.manager [^[[01;36mreq-5bcd4a69-13e3-4ec4-9a24-f665e61759e9 ^[[00;36mctx_rally_06cccf92bf84487293a6cf350e10721c_user_0 ctx_rally_d2f4c584-4152-4e5a-97ab-d2e9e019b22b_tenant_12^[[01;31m] ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[01;31mInstance failed to spawn^[[00m
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00mTraceback (most recent call last):
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2182, in _build_resources
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    yield resources
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 2029, in _build_and_run_instance
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    block_device_info=block_device_info)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2517, in spawn
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    block_device_info=block_device_info)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4631, in _create_domain_and_network
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    xml, pause=pause, power_on=power_on)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4561, in _create_domain
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    guest.launch(pause=pause)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 142, in launch
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    self._encoded_xml, errors='ignore')
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    six.reraise(self.type_, self.value, self.tb)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 137, in launch
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    return self._domain.createWithFlags(flags)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    result = proxy_call(self._autowrap, f, *args, **kwargs)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    rv = execute(f, *args, **kwargs)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    six.reraise(c, e, tb)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    rv = meth(*args, **kwargs)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00m    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  ^[[01;31m2016-01-13 21:41:04.550 TRACE nova.compute.manager ^[[01;35m[instance: a89c8f38-618e-4307-8f42-b243c900a674] ^[[00mlibvirtError: Unable to create tap device tap791b4be3-3c: Device or resource busy


  XLM copied from logs has 7 tap devices while it should have 4 only
  97ab-d2e9e019b22b_tenant_12^[[01;31m] ^[[01;35m^[[01;31mError launching a defined domain with XML: <domain type='kvm'>
    <name>instance-00000072</name>
    <uuid>a89c8f38-618e-4307-8f42-b243c900a674</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="13.0.0"/>
        <nova:name>rally_novaserver_j9L1M5GDhn3EgUq1_0</nova:name>
        <nova:creationTime>2016-01-13 16:11:03</nova:creationTime>
        <nova:flavor name="m1.tiny">
          <nova:memory>512</nova:memory>
          <nova:disk>1</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>1</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="0e158a8b454b42488e451e585ccc1366">ctx_rally_06cccf92bf84487293a6cf350e10721c_user_0</nova:user>
          <nova:project uuid="06cccf92bf84487293a6cf350e10721c">ctx_rally_d2f4c584-4152-4e5a-97ab-d2e9e019b22b_tenant_12</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="cd8e551c-288e-4585-a0da-fc3e43852ed4"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>524288</memory>
    <currentMemory unit='KiB'>524288</currentMemory>
    <vcpu placement='static'>1</vcpu>
    <cputune>
      <shares>1024</shares>
    </cputune>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>OpenStack Foundation</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>13.0.0</entry>
        <entry name='serial'>4237a912-c226-57d7-55d8-484a848739d1</entry>
        <entry name='uuid'>a89c8f38-618e-4307-8f42-b243c900a674</entry>
        <entry name='family'>Virtual Machine</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
      <kernel>/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/kernel</kernel>
      <initrd>/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/ramdisk</initrd>
      <cmdline>root=/dev/vda console=tty0 console=ttyS0</cmdline>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu>
      <topology sockets='1' cores='1' threads='1'/>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/bin/kvm-spice</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/disk'/>
        <target dev='vda' bus='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
      </disk>
      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw' cache='none'/>
        <source file='/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/disk.config'/>
        <target dev='hdd' bus='ide'/>
        <readonly/>
        <address type='drive' controller='0' bus='1' target='0' unit='1'/>
      </disk>
      <controller type='usb' index='0'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'/>
      <controller type='ide' index='0'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
      </controller>
      <interface type='bridge'>
        <mac address='fa:16:3e:36:5a:f7'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='068a041a-eded-4452-bfb8-931012f4f213'/>
        </virtualport>
        <target dev='tap068a041a-ed'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:0f:96:8b'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='791b4be3-3c03-4cba-bb0e-e9744aac862b'/>
        </virtualport>
        <target dev='tap791b4be3-3c'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:2d:46:45'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='ef9f910d-683a-4297-bcca-497026cd92dc'/>
        </virtualport>
        <target dev='tapef9f910d-68'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:0f:96:8b'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='791b4be3-3c03-4cba-bb0e-e9744aac862b'/>
        </virtualport>
        <target dev='tap791b4be3-3c'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:36:5a:f7'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='068a041a-eded-4452-bfb8-931012f4f213'/>
        </virtualport>
        <target dev='tap068a041a-ed'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:2d:46:45'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='ef9f910d-683a-4297-bcca-497026cd92dc'/>
        </virtualport>
        <target dev='tapef9f910d-68'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='fa:16:3e:0f:60:ef'/>
        <source bridge='nsx-managed'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='cdedc99a-8380-4908-9569-ad8c3e6550b4'/>
        </virtualport>
        <target dev='tapcdedc99a-83'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
      </interface>
      <serial type='file'>
        <source path='/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/console.log'/>
        <target port='0'/>
      </serial>
      <serial type='pty'>
        <target port='1'/>
      </serial>
      <console type='file'>
        <source path='/opt/stack/data/nova/instances/a89c8f38-618e-4307-8f42-b243c900a674/console.log'/>
        <target type='serial' port='0'/>
      </console>
      <input type='mouse' bus='ps2'/>
      <input type='keyboard' bus='ps2'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='10.112.203.208' keymap='en-us'>
        <listen type='address' address='10.112.203.208'/>
      </graphics>
      <video>
        <model type='cirrus' vram='9216' heads='1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
        <stats period='10'/>
      </memballoon>
    </devices>
  </domain>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1533572/+subscriptions
References

[Bug 1533572] [NEW] Failed to create vm on kvm while boot with multiple nics
From: Saurabh, 2016-01-13