yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1864347] Re: libvirt: start an instance failed lead to xml undefined

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <balazs.gibizer@xxxxxxxx>
Date: Wed, 15 Apr 2020 12:09:22 -0000
Reply-to: Bug 1864347 <1864347@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
When you say "the failed action lead to the instance is undefined which is unacceptable (we accept fail to start, but not the VM undefine)" do you mean that your problem is that nova undefined the domain in libvirt?
If yes, then how does this effects you as a nova user? You also mentioned that the VM is defined again later so I guess nova handles the situation properly. 

This is what I tried.
* I booted a cirros instance in my devstack with nova
* I stopped the instance with nova stop
* I created memory pressure in the devstack
* I tried to start the instance with nova start. It failed with a similar qemu error than in your case

2020-04-15T12:04:31.672553Z qemu-system-x86_64: cannot set up guest
memory 'pc.ram': Cannot allocate memory

* I did observe that nova undefined the domain in libvirt
* I removed the memory pressure from the devstack
* I started the instance again. It is started up properly.

Based on this I don't see any user facing issue with the current nova
behavior. So I'm setting the bug state to Opinion. Feel free to respond
here and set the bug state back to New


Also it seems that your link https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3059 points to some seemingly unrelated code path. Could you please use permalinks from github (click on a code line, then click on the ... appearing next the line number and select copy permalink)

** Changed in: nova
       Status: New => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1864347

Title:
  libvirt: start an instance failed lead to xml undefined

Status in OpenStack Compute (nova):
  Opinion

Bug description:
  we have an instance running on host (32G mem) and the VM itself is 24G mem
  when the host is filled with other VM consumption and we start the VM with 24G mem, it failed with memory not enough, however, the failed action lead to the instance is undefined which is unacceptable (we accept fail to start, but not the VM undefine)

  checked the code
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3059
  will undefine the VM, then 
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3101
  will define the VM again, so when the VM creation at 3101 has some problem (just like the call stack below, it will lead to VM undefined...)

  note the following logic is from Stein but the logic seems same?

  
  2020-02-20 13:14:29.534 8686 ERROR nova.virt.libvirt.driver [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Failed to start libvirt guest: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:29.911 8686 INFO os_vif [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:de:8a:3f,bridge_name='qbr219907f5-f9',has_traffic_filtering=True,id=219907f5-f9e5-4bbf-bf2d-e8b3de4af072,network=Network(14e7db65-1297-4c8f-8928-a1764272c276),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap219907f5-f9')
  2020-02-20 13:14:30.044 8686 INFO nova.compute.manager [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] [instance: e087ba59-10e2-4930-978b-5c951b6cc996] Successfully reverted task state from powering-on on failure for instance.
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server [req-0f8347ed-0191-4477-9cec-c5b37050e3e7 bfb0bf8fc2fe44908e638ab012266f0c 0dfede6c1a4b4bef8ce8c1936c5a418c - default default] Exception during message handling: libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 187, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1418, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2939, in start_instance
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2909, in _power_on
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     block_device_info)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2975, in power_on
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2865, in _hard_reboot
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5717, in _create_domain_and_network
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5686, in _create_domain_and_network
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5621, in _create_domain
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-20T05:14:29.061291Z qemu-kvm: cannot set up guest memory 's390.ram': Cannot allocate memory
  2020-02-20 13:14:30.049 8686 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1864347/+subscriptions
References

[Bug 1864347] [NEW] libvirt: start an instance failed lead to xml undefined
From: jichenjc, 2020-02-23