yahoo-eng-team team mailing list archive

[OpenStack Infra <1512955@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/241127
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ecd3eb7c945a7b7939c29617c2000a30a48c55dd
Submitter: Jenkins
Branch:    master

commit ecd3eb7c945a7b7939c29617c2000a30a48c55dd
Author: Huan Xie <huan.xie@xxxxxxxxxx>
Date:   Tue Nov 3 06:47:28 2015 +0000

    XenAPI:Resolve Nova/Neutron race condition
    
    When booting an instance, nova and neutron has race condition because
    nova don't know whether vif(port) is ready in neutron. There is a
    mechenism that letting neutron notify nova when port status changed
    from down to active. This fix is for xen driver to add usage of this
    event notification to avoid race condition
    
    Closes-Bug: #1512955
    
    Change-Id: I77be3bb728db72e01701c94ee292fa0f237358ed


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1512955

Title:
  Race condition in nova/neutron when booting instance with XenAPI
  driver

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  1. My environment is:
      Xenserver 6.5
      OpenStack latest master branch
      Neutron network with ML2 plugin, OVS driver

  2. When I ran tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops it will always fail.
  It failed at finding the new created instance's port before assign a floating ip to this port.

  Relevant log file of tempest:
  tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
  -------------------------------------------------------------------------------------------------------------------------------------------------
  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/test.py", line 127, in wrapper
          return f(self, *func_args, **func_kwargs)
        File "tempest/scenario/test_network_basic_ops.py", line 398, in test_network_basic_ops
          self._setup_network_and_servers()
        File "tempest/scenario/test_network_basic_ops.py", line 123, in _setup_network_and_servers
          floating_ip = self.create_floating_ip(server)
        File "tempest/scenario/manager.py", line 774, in create_floating_ip
          port_id, ip4 = self._get_server_port_id_and_ip4(thing)
        File "tempest/scenario/manager.py", line 755, in _get_server_port_id_and_ip4
          % port_map)
        File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
          self.assertThat(observed, matcher, message)
        File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
          raise mismatch_error
      testtools.matchers._impl.MismatchError: 0 != 1: Found multiple IPv4 addresses: []. Unable to determine which port to target.
  ----------------------------------------------------------------------------------------------------------------------------------------------------

  3. This is a failure that can be reproduced each time as long as we
  use xen and neutron.

  4. I tried investagating with this problem, it dues to nova/neutron
  race condition when booting an instance under xen driver, since xen
  driver doesn't handle neutron's "network-vif-plugged" notification
  event.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1512955/+subscriptions