← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #64315

[Bug 1691850] Re: Nova doesn't cleanup port when instance launch fails due to virt issues

  Thread Previous • Date Previous • • Thread Next 
*** This bug is a duplicate of bug 1648840 ***
    https://bugs.launchpad.net/bugs/1648840

This sounds like a duplicate of bug 1648840 which was fixed and
backported to stable/newton and release in 14.0.3:

https://review.openstack.org/#/c/409706/

stable/mitaka is end of life upstream though so we don't have that patch
in mitaka.

Note that that change introduced a regression which was fixed by this
backport:

https://review.openstack.org/#/c/428671/

Which went into 14.0.4.

** This bug has been marked a duplicate of bug 1648840
   libvirt driver leaves interface residue after failed start

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1691850

Title:
  Nova doesn't cleanup port when instance launch fails due to virt
  issues

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Nova doesn't cleanup port when instance launch fails on a hypervisor due to virt issues, instance is automatically rescheduled to a different hypervisor which results in duplicate ports for the instance.

  Steps to reproduce
  ==================
  A chronological list of steps which will bring off the
  issue you noticed:
  * I booted a VM (openstack server create --image cirros --flavor m1.tiny --nic net-id=network-id test-vm)
  * then instance was scheduled on hypervisor h1
  * then instance launch failed due to virt errors on h1
  * then nova did not cleanup the port while cleaning up the failed instance
  * then instance was rescheduled on hypervisor h2
  * then instance booted successfully
  * then I did openstack server list
  * then instance showed up with two IPs
  * then I did openstack port list
  * then there were two ports for the same instance

  Expected result
  ===============
  There should only be one port for an instance

  Actual result
  =============
  There were two ports for the instance

  Environment
  ===========
  1. Exact version of OpenStack you are running.
  openstack-nova-common.noarch         1:13.1.2-1.el7
  openstack-nova-compute.noarch        1:13.1.2-1.el7
  python-nova.noarch                   1:13.1.2-1.el7
  python2-novaclient.noarch            1:3.3.2-1.el7

  2. Which hypervisor did you use?
     Libvirt + KVM
     libvirt.x86_64                       1.2.17-13.el7_2.5
     qemu-kvm.x86_64                      10:1.5.3-105.el7_2.7
     CentOS Linux release 7.2.1511 (Core)

     What's the version of that?

  3. Which networking type did you use?
     neutron with OpenContrail

  Logs & Configs
  ==============

  Instance launch failure exceptions from the compute node:
  =========================================================

  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [req-c5182216-e2e9-4f8b-a7d4-373f60ac080d e54c8e8fcf4e481d9c108b47fb51834b ac7cde515a7c426986ec1c114fffe02f - - -] [instance: d58c9e56-8610-45f7-be19-2c6649070557] Instance failed to spawn
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] Traceback (most recent call last):
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] yield resources
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2779, in spawn
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4923, in _create_domain_and_network
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] xml, pause=pause, power_on=power_on)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4849, in _create_domain
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] guest = libvirt_guest.Guest.create(xml, self._host)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 127, in create
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] encodeutils.safe_decode(xml))
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] self.force_reraise()
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(self.type_, self.value, self.tb)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 123, in create
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] domain = host.write_instance_config(xml)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 981, in write_instance_config
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] return self.get_connection().defineXML(xml)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = execute(f, *args, **kwargs)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(c, e, tb)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = meth(*args, **kwargs)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3629, in defineXML
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
  2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] libvirtError: invalid argument: could not find capabilities for domaintype=kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1691850/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next