← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #19186

[Bug 1357599] [NEW] race condition with neutron in nova migrate code

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The tempest test that does a resize on the instance from time to time
fails with a neutron virtual interface timeout error. The reason why
this is occurring is because resize_instance() calls:

            disk_info = self.driver.migrate_disk_and_power_off(
                    context, instance, migration.dest_host,
                    instance_type, network_info,
                    block_device_info)

which calls destory() which unplugs the vifs(). Then,

            self.driver.finish_migration(context, migration, instance,
                                         disk_info,
                                         network_info,
                                         image, resize_instance,
                                         block_device_info, power_on)

is called which expects a vif_plugged event. Since this happens on the
same host the neutron agent is able to detect that the vif was unplugged
then plugged because it happens so fast.  To fix this we should check if
we are migrating to the same host if we are we should not expect to get
an event.


8d1] Setting instance vm_state to ERROR
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1] Traceback (most recent call last):
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3714, in finish_resize
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     disk_info, image)
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3682, in _finish_resize
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     old_instance_type, sys_meta)
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     six.reraise(self.type_, self.value, self.tb)
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3677, in _finish_resize
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     block_device_info, power_on)
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5302, in finish_migration
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     block_device_info, power_on)
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in _create_domain_and_network
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     raise exception.VirtualInterfaceCreateException()
2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1] VirtualInterfaceCreateException: Virtual Interface creation failed

** Affects: nova
     Importance: High
     Assignee: Aaron Rosen (arosen)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Aaron Rosen (arosen)

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1357599

Title:
  race condition with neutron in nova migrate code

Status in OpenStack Compute (Nova):
  New

Bug description:
  The tempest test that does a resize on the instance from time to time
  fails with a neutron virtual interface timeout error. The reason why
  this is occurring is because resize_instance() calls:

              disk_info = self.driver.migrate_disk_and_power_off(
                      context, instance, migration.dest_host,
                      instance_type, network_info,
                      block_device_info)

  which calls destory() which unplugs the vifs(). Then,

              self.driver.finish_migration(context, migration, instance,
                                           disk_info,
                                           network_info,
                                           image, resize_instance,
                                           block_device_info, power_on)

  is called which expects a vif_plugged event. Since this happens on the
  same host the neutron agent is able to detect that the vif was
  unplugged then plugged because it happens so fast.  To fix this we
  should check if we are migrating to the same host if we are we should
  not expect to get an event.


  8d1] Setting instance vm_state to ERROR
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1] Traceback (most recent call last):
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3714, in finish_resize
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     disk_info, image)
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3682, in _finish_resize
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     old_instance_type, sys_meta)
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     six.reraise(self.type_, self.value, self.tb)
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3677, in _finish_resize
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     block_device_info, power_on)
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5302, in finish_migration
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     block_device_info, power_on)
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3792, in _create_domain_and_network
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1]     raise exception.VirtualInterfaceCreateException()
  2014-08-14 00:03:58.010 1276 TRACE nova.compute.manager [instance: dca468e4-d26f-4ae2-a522-7d02ef7c98d1] VirtualInterfaceCreateException: Virtual Interface creation failed

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1357599/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next