yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712460] Re: nova destroys wrong instance if trying to create a new instance with the same name

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Mon, 28 Aug 2017 14:48:05 -0000
Reply-to: Bug 1712460 <1712460@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/pike
       Status: New => Confirmed

** Changed in: nova/pike
   Importance: Undecided => High

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Changed in: nova/ocata
       Status: New => Confirmed

** Changed in: nova/ocata
     Assignee: (unassigned) => Sean Dague (sdague)

** Changed in: nova/pike
     Assignee: (unassigned) => Sean Dague (sdague)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1712460

Title:
  nova destroys wrong instance if trying to create a new instance with
  the same name

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) ocata series:
  Confirmed
Status in OpenStack Compute (nova) pike series:
  Confirmed

Bug description:
  
  Description
  ===========
  If trying to create a new VM instance(let's call vm1) with the same instance name
  on the same compute host, old existing VM instance(vm1) is destroyed. And openstack
  does not intend such situation so the status of old instance(vm1) is 'ACTIVE'

  
  Steps to reproduce
  ==================
  A chronological list of steps which will bring off the
  issue you noticed:
  1. create one VM.
  2. check the VM is created.
  3. create another VM on the same host with the same name of the created VM in step 1
    * In order to make same named instance, we can change nova config like below
       instance_name_template = %(hostname)s 
  4. check the VM exist. openstack has active instance but real VM is not.
  :
  [controller]# nova list
  +--------------------------------------+--------+--------+------------+-------------+----------------------+
  | ID                                   | Name   | Status | Task State | Power State | Networks             |
  +--------------------------------------+--------+--------+------------+-------------+----------------------+
  | 8595078e-1635-47c3-a027-78436a17f6d4 | testvm | ERROR  | -          | NOSTATE     | demo-net=192.168.0.3 |
  | a2239b07-9811-444f-b77e-96756046e8de | testvm | ACTIVE | -          | Running     | demo-net=192.168.0.7 |
  +--------------------------------------+--------+--------+------------+-------------+----------------------+
  [compute1]# virsh list
   Id    Name                           State
  ----------------------------------------------------

  Expected result
  ===============
  old VM instance should be running.

  Actual result
  =============
  There is no running VM.

  Environment
  ===========
  1. Exact version of OpenStack you are running. 
  Tested on Ocata

  2. Which hypervisor did you use?
  Libvirt + KVM

  
  Logs & Configs
  ==============
  # grep instance_name_template /etc/nova/nova.conf 
  #instance_name_template=instance-%08x
  instance_name_template = %(hostname)s

  <nova log>
  2017-08-21 04:13:11.692 27134 ERROR nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Failed to start libvirt guest
  2017-08-21 04:13:11.693 27134 INFO os_vif [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:ad:3d:09,bridge_name='brq0f056785-2d',has_traffic_filtering=True,id=a74d54bd-04ab-42e1-bbea-2417f487825c,network=Network(0f056785-2d0a-4742-9517-570dbc427e16),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapa74d54bd-04')
  2017-08-21 04:13:11.704 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deleting instance files /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del
  2017-08-21 04:13:11.705 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deletion of /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del complete
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Instance failed to spawn
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Traceback (most recent call last):
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     yield resources
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1931, in _build_and_run_instance
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     block_device_info=block_device_info)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2717, in spawn
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     destroy_disks_on_failure=True)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5179, in _create_domain_and_network
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     destroy_disks_on_failure)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     self.force_reraise()
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     six.reraise(self.type_, self.value, self.tb)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5151, in _create_domain_and_network
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     post_xml_callback=post_xml_callback)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5062, in _create_domain
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     guest = libvirt_guest.Guest.create(xml, self._host)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 130, in create
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     encodeutils.safe_decode(xml))
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     self.force_reraise()
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     six.reraise(self.type_, self.value, self.tb)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 126, in create
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     guest = host.write_instance_config(xml)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 835, in write_instance_config
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     domain = self.get_connection().defineXML(xml)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     rv = execute(f, *args, **kwargs)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     six.reraise(c, e, tb)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     rv = meth(*args, **kwargs)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3585, in defineXML
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]     if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] libvirtError: operation failed: domain 'testvm' already exists with uuid a2239b07-9811-444f-b77e-96756046e8de
  2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] 
  2017-08-21 04:13:14.108 27134 INFO nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Terminating instance
  2017-08-21 04:13:14.109 27134 WARNING nova.image.glance [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] No protocol specified in for api_server 'http://128.80.0.6:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)
  2017-08-21 04:13:14.329 27134 INFO nova.virt.libvirt.driver [-] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Instance destroyed successfully.
  2017-08-21 04:13:14.332 27134 INFO os_vif [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:ad:3d:09,bridge_name='brq0f056785-2d',has_traffic_filtering=True,id=a74d54bd-04ab-42e1-bbea-2417f487825c,network=Network(0f056785-2d0a-4742-9517-570dbc427e16),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapa74d54bd-04')
  2017-08-21 04:13:14.359 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deletion of /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del complete
  2017-08-21 04:13:18.867 27134 WARNING nova.compute.manager [req-eef35b0d-a9a9-44c3-a21e-2cb1fff74224 2d6b9a3227854bfda6ffaca4e1d231b2 843b7144916f44c4adf4cd6290febb51 - - -] [instance: a2239b07-9811-444f-b77e-96756046e8de] Received unexpected event network-vif-unplugged-96e65960-54da-4d41-90b8-23243864d5ba for instance
  2017-08-21 04:13:19.503 27134 INFO nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Took 5.39 seconds to destroy the instance on the hypervisor.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1712460/+subscriptions
References

[Bug 1712460] [NEW] nova destroys wrong instance if trying to create a new instance with the same name
From: Wonil Choi, 2017-08-23