← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #13648

[Bug 1308544] [NEW] libvirt: Trying to delete a non-existing vif raises an exception

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

If an instance fails during its network creation (for example if the
network-vif-plugged event doesn't arrive in time) a subsequent delete
will also fail when it tries to delete the vif, leaving the instance in
a Error(deleting) state.

This can be avoided by including the "--if-exists" option to the
ovs=vsctl command.

Example of stack trace:

 2014-04-16 12:28:51.949 AUDIT nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Terminating instance
2014-04-16 12:28:52.309 ERROR nova.virt.libvirt.driver [-] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] During wait destroy, instance disappeared.
2014-04-16 12:28:52.407 ERROR nova.network.linux_net [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] Unable to execute ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']. Exception: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo67a96e96-10
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: no port named qvo67a96e96-10\n'
2014-04-16 12:28:52.573 ERROR nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Setting instance vm_state to ERROR
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Traceback (most recent call last):
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2261, in do_terminate_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self._delete_instance(context, instance, bdms, quotas)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/hooks.py", line 103, in inner
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     rv = f(*args, **kwargs)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2231, in _delete_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     quotas.rollback()
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     six.reraise(self.type_, self.value, self.tb)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2203, in _delete_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self._shutdown_instance(context, db_inst, bdms)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2145, in _shutdown_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     requested_networks)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     six.reraise(self.type_, self.value, self.tb)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2135, in _shutdown_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     block_device_info)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 955, in destroy
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     destroy_disks)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 991, in cleanup
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_vifs(instance, network_info)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 863, in unplug_vifs
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.vif_driver.unplug(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_ovs(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_ovs_hybrid(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     v2_name)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/network/linux_net.py", line 1318, in delete_ovs_vif_port
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     _ovs_vsctl(['del-port', bridge, dev])
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     raise exception.AgentError(method=full_args)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']

** Affects: nova
     Importance: Undecided
     Assignee: Phil Day (philip-day)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Phil Day (philip-day)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1308544

Title:
  libvirt: Trying to delete a non-existing vif raises an exception

Status in OpenStack Compute (Nova):
  New

Bug description:
  If an instance fails during its network creation (for example if the
  network-vif-plugged event doesn't arrive in time) a subsequent delete
  will also fail when it tries to delete the vif, leaving the instance
  in a Error(deleting) state.

  This can be avoided by including the "--if-exists" option to the
  ovs=vsctl command.

  Example of stack trace:

   2014-04-16 12:28:51.949 AUDIT nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Terminating instance
  2014-04-16 12:28:52.309 ERROR nova.virt.libvirt.driver [-] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] During wait destroy, instance disappeared.
  2014-04-16 12:28:52.407 ERROR nova.network.linux_net [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] Unable to execute ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']. Exception: Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo67a96e96-10
  Exit code: 1
  Stdout: ''
  Stderr: 'ovs-vsctl: no port named qvo67a96e96-10\n'
  2014-04-16 12:28:52.573 ERROR nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Setting instance vm_state to ERROR
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Traceback (most recent call last):
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2261, in do_terminate_instance
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self._delete_instance(context, instance, bdms, quotas)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/hooks.py", line 103, in inner
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     rv = f(*args, **kwargs)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2231, in _delete_instance
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     quotas.rollback()
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     six.reraise(self.type_, self.value, self.tb)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2203, in _delete_instance
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self._shutdown_instance(context, db_inst, bdms)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2145, in _shutdown_instance
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     requested_networks)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     six.reraise(self.type_, self.value, self.tb)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/compute/manager.py", line 2135, in _shutdown_instance
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     block_device_info)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 955, in destroy
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     destroy_disks)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 991, in cleanup
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_vifs(instance, network_info)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 863, in unplug_vifs
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.vif_driver.unplug(instance, vif)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_ovs(instance, vif)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     self.unplug_ovs_hybrid(instance, vif)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     v2_name)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/network/linux_net.py", line 1318, in delete_ovs_vif_port
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     _ovs_vsctl(['del-port', bridge, dev])
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]   File "/mnt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f]     raise exception.AgentError(method=full_args)
  2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1308544/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next