← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #78232

[Bug 1826701] [NEW] the rbd backend root disks of the virtual machine will be cleared if launch failed when do evacuate

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I found that some of the instances could not get started after do
evacuate failed, then I check the nova-compute log [1], that point out
instance root disk volume 'c23c04c9-2a8b-492e-8130-99aafa30b563_disk'
can not be found in ceph compute pool, look back to evacuate failed log
[2],  there was an error occur 'libvirtError: Failed to create
controller memory for group: No space left on device' when do launch.

check the nova code, when this error occured, the evacuate action in nova-compute function call stack is:
 https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/compute/manager.py#L3044
nova.compute.mananger.ComputeManager.rebuild_instance -> _do_rebuild_instance_with_claim -> _do_rebuild_instance -> _rebuild_default_impl -> driver.spawn

https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/virt/libvirt/driver.py#L3154
nova.virt.libvirt.driver.LibvirtDriver.spawn -> _create_domain_and_network ->  _cleanup_failed_start -> cleanup -> _cleanup_rbd

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L360
nova.virt.libvirt.storage.rbd_utils.RBDDriver.cleanup_volumes -> _destroy_volume

this logic make the instance root disk with rbd image backend was clean
up in ceph, and nerver get started again, and even make the data lost,
is this reasonable?


[1] instance start failed log
2019-04-26 11:44:45.298 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-26 11:44:45.403 46085 WARNING nova.virt.osinfo [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-26 11:44:45.424 46085 INFO os_vif [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
2019-04-26 11:44:45.516 46085 WARNING nova.virt.osinfo [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-26 11:44:45.536 46085 INFO os_vif [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:62:d6:59,bridge_name='br-int',has_traffic_filtering=True,id=cdd730dc-c806-424f-8d7a-96253e9a72b1,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapcdd730dc-c8')
2019-04-26 11:44:45.666 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-26 11:44:45.684 46085 INFO os_vif [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:c9:7d:69,bridge_name='br-int',has_traffic_filtering=True,id=06510302-8d87-4d3e-90e1-2a7bcbb14f6b,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap06510302-8d')
2019-04-26 11:44:46.000 46085 ERROR nova.virt.libvirt.guest [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-000000bc</name>
  <uuid>c23c04c9-2a8b-492e-8130-99aafa30b563</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="0.0.1"/>
      <nova:name>SIIT-SL-ES1</nova:name>
      <nova:creationTime>2019-04-26 03:44:45</nova:creationTime>
      <nova:flavor name="16-32768-500">
        <nova:memory>32768</nova:memory>
        <nova:disk>500</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>16</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="5a55fb96f12e42f1a8402faf4593eb4a">admin</nova:user>
        <nova:project uuid="7bcb8c85147e42e99a4f4687179a2203">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="ea6bffc1-b9d0-48cc-9842-bf69fbfe05f9"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static' cpuset='0-11,24-35'>16</vcpu>
  <cputune>
    <shares>16384</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>0.0.1</entry>
      <entry name='serial'>c792c755-66d4-4ef9-b7db-dd6bb7ff89f8</entry>
      <entry name='uuid'>c23c04c9-2a8b-492e-8130-99aafa30b563</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='16' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <auth username='admin'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk'>
        <host name='192.168.15.2' port='6789'/>
        <host name='192.168.15.3' port='6789'/>
        <host name='192.168.15.4' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='admin'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='volumes/7b130e4a-1025-4d6b-b7f5-7476d86f8f32'>
        <host name='192.168.15.2' port='6789'/>
        <host name='192.168.15.3' port='6789'/>
        <host name='192.168.15.4' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <serial>7b130e4a-1025-4d6b-b7f5-7476d86f8f32</serial>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:69:df:d7'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='89b882ea-15f0-4e2c-b3b1-a515e3a29f52'/>
      </virtualport>
      <target dev='tap89b882ea-15'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>


2019-04-26 11:44:46.001 46085 ERROR nova.virt.libvirt.driver [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Failed to start libvirt guest
2019-04-26 11:44:46.002 46085 INFO os_vif [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
2019-04-26 11:44:46.148 46085 INFO nova.compute.manager [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Successfully reverted task state from powering-on on failure for instance.
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Exception during message handling
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 191, in decorated_function
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 160, in decorated_function
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 219, in decorated_function
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 207, in decorated_function
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2487, in start_instance
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2457, in _power_on
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2490, in power_on
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2379, in _hard_reboot
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5014, in _create_domain_and_network
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4986, in _create_domain_and_network
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4899, in _create_domain
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-26T03:44:45.683489Z qemu-kvm: -chardev pty,id=charserial1: char device redirected to /dev/pts/1 (label charserial1)
2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server 2019-04-26T03:44:45.737073Z qemu-kvm: -drive file=rbd:compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk:id=admin:auth_supported=cephx;none:mon_host=192.168.15.2:6789;192.168.15.3:6789;192.168.15.4:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: error reading header from c23c04c9-2a8b-492e-8130-99aafa30b563_disk: No such file or directory


[2] evcaute failed log
2019-04-25 19:09:26.904 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Rebuilding instance
2019-04-25 19:09:27.069 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Attempting claim: memory 32768 MB, disk 500 GB, vcpus 16 CPU
2019-04-25 19:09:27.070 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total memory: 261758 MB, used: 226304.00 MB
2019-04-25 19:09:27.070 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] memory limit not specified, defaulting to unlimited
2019-04-25 19:09:27.071 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total disk: 19256 GB, used: 1700.00 GB
2019-04-25 19:09:27.072 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] disk limit not specified, defaulting to unlimited
2019-04-25 19:09:27.072 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total vcpu: 36 VCPU, used: 64.00 VCPU
2019-04-25 19:09:27.073 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] vcpu limit not specified, defaulting to unlimited
2019-04-25 19:09:27.074 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Claim successful
2019-04-25 19:09:27.239 46065 INFO nova.compute.resource_tracker [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Updating from migration c23c04c9-2a8b-492e-8130-99aafa30b563
2019-04-25 19:09:27.291 46065 INFO nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Another thread already created a resource provider with the UUID 9b3ef9f5-a298-4aeb-b9ec-4cba7f06149f. Grabbing that record from the placement API.
2019-04-25 19:09:27.301 46065 WARNING nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Unable to refresh my resource provider record
2019-04-25 19:09:27.303 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] disk on shared storage, recreating using existing disk
2019-04-25 19:09:28.049 46065 INFO nova.network.neutronv2.api [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Updating port 89b882ea-15f0-4e2c-b3b1-a515e3a29f52 with attributes {'binding:profile': {}, 'binding:host_id': 'node-4.domain.tld'}
2019-04-25 19:09:28.691 46065 WARNING nova.virt.osinfo [req-b6fc360d-b390-498b-9ee7-c5847a937258 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-25 19:09:29.655 46065 INFO nova.virt.block_device [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Booting with blank volume at /dev/vdb
2019-04-25 19:09:35.208 46065 WARNING nova.virt.osinfo [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
2019-04-25 19:09:35.238 46065 INFO os_vif [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Successfully plugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
2019-04-25 19:09:35.793 46065 ERROR nova.virt.libvirt.guest [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-000000bc</name>
  <uuid>c23c04c9-2a8b-492e-8130-99aafa30b563</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="0.0.1"/>
      <nova:name>SIIT-SL-ES1</nova:name>
      <nova:creationTime>2019-04-25 11:09:34</nova:creationTime>
      <nova:flavor name="16-32768-500">
        <nova:memory>32768</nova:memory>
        <nova:disk>500</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>16</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="2db8df2b45f0473989beaae02155485b">nova</nova:user>
        <nova:project uuid="369f9e4d4b3f4285b30b8705e2badcf8">service</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="ea6bffc1-b9d0-48cc-9842-bf69fbfe05f9"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static' cpuset='0-11,24-47'>16</vcpu>
  <cputune>
    <shares>16384</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>0.0.1</entry>
      <entry name='serial'>f306edb9-c4c1-48a9-ae24-0720ff168a8c</entry>
      <entry name='uuid'>c23c04c9-2a8b-492e-8130-99aafa30b563</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='16' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <auth username='admin'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk'>
        <host name='192.168.15.2' port='6789'/>
        <host name='192.168.15.3' port='6789'/>
        <host name='192.168.15.4' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='admin'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='volumes/7b130e4a-1025-4d6b-b7f5-7476d86f8f32'>
        <host name='192.168.15.2' port='6789'/>
        <host name='192.168.15.3' port='6789'/>
        <host name='192.168.15.4' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <serial>7b130e4a-1025-4d6b-b7f5-7476d86f8f32</serial>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:69:df:d7'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='89b882ea-15f0-4e2c-b3b1-a515e3a29f52'/>
      </virtualport>
      <target dev='tap89b882ea-15'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>


2019-04-25 19:09:35.795 46065 ERROR nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Failed to start libvirt guest
2019-04-25 19:09:35.796 46065 INFO os_vif [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Successfully unplugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
2019-04-25 19:09:50.129 46065 INFO nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Deleting instance files /var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563_del
2019-04-25 19:09:50.130 46065 INFO nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Deletion of /var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563_del complete
2019-04-25 19:09:50.306 46065 INFO nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Another thread already created a resource provider with the UUID 9b3ef9f5-a298-4aeb-b9ec-4cba7f06149f. Grabbing that record from the placement API.
2019-04-25 19:09:50.320 46065 WARNING nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Unable to refresh my resource provider record
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Setting instance vm_state to ERROR
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Traceback (most recent call last):
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6667, in _error_out_instance_on_exception
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2756, in _do_rebuild_instance_with_claim
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2871, in _do_rebuild_instance
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2624, in _rebuild_default_impl
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2672, in spawn
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5014, in _create_domain_and_network
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     self.force_reraise()
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(self.type_, self.value, self.tb)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4986, in _create_domain_and_network
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4899, in _create_domain
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     self.force_reraise()
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(self.type_, self.value, self.tb)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     rv = execute(f, *args, **kwargs)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(c, e, tb)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     rv = meth(*args, **kwargs)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] libvirtError: Failed to create controller memory for group: No space left on device
2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] 
2019-04-25 19:09:50.620 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Successfully reverted task state from rebuild_spawning on failure for instance.
2019-04-25 19:09:50.791 46065 ERROR oslo_messaging.rpc.server [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1826701

Title:
  the rbd backend root disks of the virtual machine will be cleared if
  launch failed when do evacuate

Status in OpenStack Compute (nova):
  New

Bug description:
  I found that some of the instances could not get started after do
  evacuate failed, then I check the nova-compute log [1], that point out
  instance root disk volume 'c23c04c9-2a8b-492e-8130-99aafa30b563_disk'
  can not be found in ceph compute pool, look back to evacuate failed
  log [2],  there was an error occur 'libvirtError: Failed to create
  controller memory for group: No space left on device' when do launch.

  check the nova code, when this error occured, the evacuate action in nova-compute function call stack is:
   https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/compute/manager.py#L3044
  nova.compute.mananger.ComputeManager.rebuild_instance -> _do_rebuild_instance_with_claim -> _do_rebuild_instance -> _rebuild_default_impl -> driver.spawn

  https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/virt/libvirt/driver.py#L3154
  nova.virt.libvirt.driver.LibvirtDriver.spawn -> _create_domain_and_network ->  _cleanup_failed_start -> cleanup -> _cleanup_rbd

  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L360
  nova.virt.libvirt.storage.rbd_utils.RBDDriver.cleanup_volumes -> _destroy_volume

  this logic make the instance root disk with rbd image backend was
  clean up in ceph, and nerver get started again, and even make the data
  lost, is this reasonable?

  
  [1] instance start failed log
  2019-04-26 11:44:45.298 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-26 11:44:45.403 46085 WARNING nova.virt.osinfo [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-26 11:44:45.424 46085 INFO os_vif [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
  2019-04-26 11:44:45.516 46085 WARNING nova.virt.osinfo [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-26 11:44:45.536 46085 INFO os_vif [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:62:d6:59,bridge_name='br-int',has_traffic_filtering=True,id=cdd730dc-c806-424f-8d7a-96253e9a72b1,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapcdd730dc-c8')
  2019-04-26 11:44:45.666 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-26 11:44:45.684 46085 INFO os_vif [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:c9:7d:69,bridge_name='br-int',has_traffic_filtering=True,id=06510302-8d87-4d3e-90e1-2a7bcbb14f6b,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap06510302-8d')
  2019-04-26 11:44:46.000 46085 ERROR nova.virt.libvirt.guest [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Error launching a defined domain with XML: <domain type='kvm'>
    <name>instance-000000bc</name>
    <uuid>c23c04c9-2a8b-492e-8130-99aafa30b563</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="0.0.1"/>
        <nova:name>SIIT-SL-ES1</nova:name>
        <nova:creationTime>2019-04-26 03:44:45</nova:creationTime>
        <nova:flavor name="16-32768-500">
          <nova:memory>32768</nova:memory>
          <nova:disk>500</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>16</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="5a55fb96f12e42f1a8402faf4593eb4a">admin</nova:user>
          <nova:project uuid="7bcb8c85147e42e99a4f4687179a2203">admin</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="ea6bffc1-b9d0-48cc-9842-bf69fbfe05f9"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>33554432</memory>
    <currentMemory unit='KiB'>33554432</currentMemory>
    <vcpu placement='static' cpuset='0-11,24-35'>16</vcpu>
    <cputune>
      <shares>16384</shares>
    </cputune>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>OpenStack Foundation</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>0.0.1</entry>
        <entry name='serial'>c792c755-66d4-4ef9-b7db-dd6bb7ff89f8</entry>
        <entry name='uuid'>c23c04c9-2a8b-492e-8130-99aafa30b563</entry>
        <entry name='family'>Virtual Machine</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='host-model' check='partial'>
      <model fallback='allow'/>
      <topology sockets='16' cores='1' threads='1'/>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none' discard='unmap'/>
        <auth username='admin'>
          <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
        </auth>
        <source protocol='rbd' name='compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk'>
          <host name='192.168.15.2' port='6789'/>
          <host name='192.168.15.3' port='6789'/>
          <host name='192.168.15.4' port='6789'/>
        </source>
        <target dev='vda' bus='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <auth username='admin'>
          <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
        </auth>
        <source protocol='rbd' name='volumes/7b130e4a-1025-4d6b-b7f5-7476d86f8f32'>
          <host name='192.168.15.2' port='6789'/>
          <host name='192.168.15.3' port='6789'/>
          <host name='192.168.15.4' port='6789'/>
        </source>
        <target dev='vdb' bus='virtio'/>
        <serial>7b130e4a-1025-4d6b-b7f5-7476d86f8f32</serial>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </disk>
      <controller type='usb' index='0' model='piix3-uhci'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'/>
      <interface type='bridge'>
        <mac address='fa:16:3e:69:df:d7'/>
        <source bridge='br-int'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='89b882ea-15f0-4e2c-b3b1-a515e3a29f52'/>
        </virtualport>
        <target dev='tap89b882ea-15'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='file'>
        <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
        <target port='0'/>
      </serial>
      <serial type='pty'>
        <target port='1'/>
      </serial>
      <console type='file'>
        <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
        <target type='serial' port='0'/>
      </console>
      <input type='tablet' bus='usb'>
        <address type='usb' bus='0' port='1'/>
      </input>
      <input type='mouse' bus='ps2'/>
      <input type='keyboard' bus='ps2'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='cirrus' vram='16384' heads='1' primary='yes'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <stats period='10'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </memballoon>
    </devices>
  </domain>

  
  2019-04-26 11:44:46.001 46085 ERROR nova.virt.libvirt.driver [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Failed to start libvirt guest
  2019-04-26 11:44:46.002 46085 INFO os_vif [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
  2019-04-26 11:44:46.148 46085 INFO nova.compute.manager [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Successfully reverted task state from powering-on on failure for instance.
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Exception during message handling
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 191, in decorated_function
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 160, in decorated_function
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 219, in decorated_function
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 207, in decorated_function
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2487, in start_instance
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2457, in _power_on
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2490, in power_on
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2379, in _hard_reboot
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5014, in _create_domain_and_network
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4986, in _create_domain_and_network
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4899, in _create_domain
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-26T03:44:45.683489Z qemu-kvm: -chardev pty,id=charserial1: char device redirected to /dev/pts/1 (label charserial1)
  2019-04-26 11:44:46.247 46085 ERROR oslo_messaging.rpc.server 2019-04-26T03:44:45.737073Z qemu-kvm: -drive file=rbd:compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk:id=admin:auth_supported=cephx;none:mon_host=192.168.15.2:6789;192.168.15.3:6789;192.168.15.4:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: error reading header from c23c04c9-2a8b-492e-8130-99aafa30b563_disk: No such file or directory

  
  [2] evcaute failed log
  2019-04-25 19:09:26.904 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Rebuilding instance
  2019-04-25 19:09:27.069 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Attempting claim: memory 32768 MB, disk 500 GB, vcpus 16 CPU
  2019-04-25 19:09:27.070 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total memory: 261758 MB, used: 226304.00 MB
  2019-04-25 19:09:27.070 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] memory limit not specified, defaulting to unlimited
  2019-04-25 19:09:27.071 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total disk: 19256 GB, used: 1700.00 GB
  2019-04-25 19:09:27.072 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] disk limit not specified, defaulting to unlimited
  2019-04-25 19:09:27.072 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Total vcpu: 36 VCPU, used: 64.00 VCPU
  2019-04-25 19:09:27.073 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] vcpu limit not specified, defaulting to unlimited
  2019-04-25 19:09:27.074 46065 INFO nova.compute.claims [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Claim successful
  2019-04-25 19:09:27.239 46065 INFO nova.compute.resource_tracker [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Updating from migration c23c04c9-2a8b-492e-8130-99aafa30b563
  2019-04-25 19:09:27.291 46065 INFO nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Another thread already created a resource provider with the UUID 9b3ef9f5-a298-4aeb-b9ec-4cba7f06149f. Grabbing that record from the placement API.
  2019-04-25 19:09:27.301 46065 WARNING nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Unable to refresh my resource provider record
  2019-04-25 19:09:27.303 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] disk on shared storage, recreating using existing disk
  2019-04-25 19:09:28.049 46065 INFO nova.network.neutronv2.api [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Updating port 89b882ea-15f0-4e2c-b3b1-a515e3a29f52 with attributes {'binding:profile': {}, 'binding:host_id': 'node-4.domain.tld'}
  2019-04-25 19:09:28.691 46065 WARNING nova.virt.osinfo [req-b6fc360d-b390-498b-9ee7-c5847a937258 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-25 19:09:29.655 46065 INFO nova.virt.block_device [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Booting with blank volume at /dev/vdb
  2019-04-25 19:09:35.208 46065 WARNING nova.virt.osinfo [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS)
  2019-04-25 19:09:35.238 46065 INFO os_vif [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Successfully plugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
  2019-04-25 19:09:35.793 46065 ERROR nova.virt.libvirt.guest [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Error launching a defined domain with XML: <domain type='kvm'>
    <name>instance-000000bc</name>
    <uuid>c23c04c9-2a8b-492e-8130-99aafa30b563</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="0.0.1"/>
        <nova:name>SIIT-SL-ES1</nova:name>
        <nova:creationTime>2019-04-25 11:09:34</nova:creationTime>
        <nova:flavor name="16-32768-500">
          <nova:memory>32768</nova:memory>
          <nova:disk>500</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>16</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="2db8df2b45f0473989beaae02155485b">nova</nova:user>
          <nova:project uuid="369f9e4d4b3f4285b30b8705e2badcf8">service</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="ea6bffc1-b9d0-48cc-9842-bf69fbfe05f9"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>33554432</memory>
    <currentMemory unit='KiB'>33554432</currentMemory>
    <vcpu placement='static' cpuset='0-11,24-47'>16</vcpu>
    <cputune>
      <shares>16384</shares>
    </cputune>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>OpenStack Foundation</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>0.0.1</entry>
        <entry name='serial'>f306edb9-c4c1-48a9-ae24-0720ff168a8c</entry>
        <entry name='uuid'>c23c04c9-2a8b-492e-8130-99aafa30b563</entry>
        <entry name='family'>Virtual Machine</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='host-model' check='partial'>
      <model fallback='allow'/>
      <topology sockets='16' cores='1' threads='1'/>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none' discard='unmap'/>
        <auth username='admin'>
          <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
        </auth>
        <source protocol='rbd' name='compute/c23c04c9-2a8b-492e-8130-99aafa30b563_disk'>
          <host name='192.168.15.2' port='6789'/>
          <host name='192.168.15.3' port='6789'/>
          <host name='192.168.15.4' port='6789'/>
        </source>
        <target dev='vda' bus='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <auth username='admin'>
          <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
        </auth>
        <source protocol='rbd' name='volumes/7b130e4a-1025-4d6b-b7f5-7476d86f8f32'>
          <host name='192.168.15.2' port='6789'/>
          <host name='192.168.15.3' port='6789'/>
          <host name='192.168.15.4' port='6789'/>
        </source>
        <target dev='vdb' bus='virtio'/>
        <serial>7b130e4a-1025-4d6b-b7f5-7476d86f8f32</serial>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </disk>
      <controller type='usb' index='0' model='piix3-uhci'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'/>
      <interface type='bridge'>
        <mac address='fa:16:3e:69:df:d7'/>
        <source bridge='br-int'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='89b882ea-15f0-4e2c-b3b1-a515e3a29f52'/>
        </virtualport>
        <target dev='tap89b882ea-15'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='file'>
        <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
        <target port='0'/>
      </serial>
      <serial type='pty'>
        <target port='1'/>
      </serial>
      <console type='file'>
        <source path='/var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563/console.log'/>
        <target type='serial' port='0'/>
      </console>
      <input type='tablet' bus='usb'>
        <address type='usb' bus='0' port='1'/>
      </input>
      <input type='mouse' bus='ps2'/>
      <input type='keyboard' bus='ps2'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='cirrus' vram='16384' heads='1' primary='yes'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <stats period='10'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </memballoon>
    </devices>
  </domain>

  
  2019-04-25 19:09:35.795 46065 ERROR nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Failed to start libvirt guest
  2019-04-25 19:09:35.796 46065 INFO os_vif [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Successfully unplugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15')
  2019-04-25 19:09:50.129 46065 INFO nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Deleting instance files /var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563_del
  2019-04-25 19:09:50.130 46065 INFO nova.virt.libvirt.driver [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Deletion of /var/lib/nova/instances/c23c04c9-2a8b-492e-8130-99aafa30b563_del complete
  2019-04-25 19:09:50.306 46065 INFO nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Another thread already created a resource provider with the UUID 9b3ef9f5-a298-4aeb-b9ec-4cba7f06149f. Grabbing that record from the placement API.
  2019-04-25 19:09:50.320 46065 WARNING nova.scheduler.client.report [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] Unable to refresh my resource provider record
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Setting instance vm_state to ERROR
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Traceback (most recent call last):
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6667, in _error_out_instance_on_exception
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2712, in rebuild_instance
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2756, in _do_rebuild_instance_with_claim
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2871, in _do_rebuild_instance
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2624, in _rebuild_default_impl
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2672, in spawn
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5014, in _create_domain_and_network
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     self.force_reraise()
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(self.type_, self.value, self.tb)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4986, in _create_domain_and_network
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4899, in _create_domain
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     self.force_reraise()
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(self.type_, self.value, self.tb)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     rv = execute(f, *args, **kwargs)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     six.reraise(c, e, tb)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     rv = meth(*args, **kwargs)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] libvirtError: Failed to create controller memory for group: No space left on device
  2019-04-25 19:09:50.375 46065 ERROR nova.compute.manager [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] 
  2019-04-25 19:09:50.620 46065 INFO nova.compute.manager [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b 369f9e4d4b3f4285b30b8705e2badcf8 - - -] [instance: c23c04c9-2a8b-492e-8130-99aafa30b563] Successfully reverted task state from rebuild_spawning on failure for instance.
  2019-04-25 19:09:50.791 46065 ERROR oslo_messaging.rpc.server [req-1f4ea446-2407-4a48-bf32-81ad2b1d90f2 2db8df2b45f0473989beaae02155485b

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1826701/+subscriptions
  Thread Previous • Date Previous • • Thread Next