yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1864347] [NEW] libvirt: start an instance failed lead to xml undefined

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: jichenjc <jichenjc@xxxxxxxxxx>
Date: Sun, 23 Feb 2020 07:56:38 -0000
Reply-to: Bug 1864347 <1864347@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

we have an instance running on host (32G mem) and the VM itself is 24G mem
when the host is filled with other VM consumption and we start the VM with 24G mem, it failed with memory not enough, however, the failed action lead to the instance is undefined which is unacceptable (we accept fail to start, but not the VM undefine)

checked the code
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3059
will undefine the VM, then 
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3101
will define the VM again, so when the VM creation at 3101 has some problem (just like the call stack below, it will lead to VM undefined...)

note the following logic is from Stein but the logic seems same?


2020-02-20 13:14:29.534 8686 ERROR nova.virt.libvirt.driver [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Failed to start libvirt guest: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
2020-02-20 13:14:29.911 8686 INFO os_vif [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:de:8a:3f,bridge_name='qbr219907f5-f9',has_traffic_filtering=True,id=219907f5-f9e5-4bbf-bf2d-e8b3de4af072,network=Network(14e7db65-1297-4c8f-8928-a1764272c276),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap219907f5-f9')
2020-02-20 13:14:30.044 8686 INFO nova.compute.manager [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Successfully reverted task state from powering-on on failure for instance.
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Exception during message handling: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1418, in decorated_function
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2939, in start_instance
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2909, in _power_on
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     block_device_info)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2975, in power_on
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2865, in _hard_reboot
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5717, in _create_domain_and_network
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5686, in _create_domain_and_network
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5621, in _create_domain
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1864347

Title:
  libvirt: start an instance failed lead to xml undefined

Status in OpenStack Compute (nova):
  New

Bug description:
  we have an instance running on host (32G mem) and the VM itself is 24G mem
  when the host is filled with other VM consumption and we start the VM with 24G mem, it failed with memory not enough, however, the failed action lead to the instance is undefined which is unacceptable (we accept fail to start, but not the VM undefine)

  checked the code
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3059
  will undefine the VM, then 
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3101
  will define the VM again, so when the VM creation at 3101 has some problem (just like the call stack below, it will lead to VM undefined...)

  note the following logic is from Stein but the logic seems same?

  
  2020-02-20 13:14:29.534 8686 ERROR nova.virt.libvirt.driver [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Failed to start libvirt guest: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:29.911 8686 INFO os_vif [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:de:8a:3f,bridge_name='qbr219907f5-f9',has_traffic_filtering=True,id=219907f5-f9e5-4bbf-bf2d-e8b3de4af072,network=Network(14e7db65-1297-4c8f-8928-a1764272c276),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap219907f5-f9')
  2020-02-20 13:14:30.044 8686 INFO nova.compute.manager [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Successfully reverted task state from powering-on on failure for instance.
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Exception during message handling: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1418, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2939, in start_instance
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2909, in _power_on
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     block_device_info)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2975, in power_on
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2865, in _hard_reboot
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5717, in _create_domain_and_network
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5686, in _create_domain_and_network
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5621, in _create_domain
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1864347/+subscriptions
Follow ups

[Bug 1864347] Re: libvirt: start an instance failed lead to xml undefined
From: Balazs Gibizer, 2020-04-15