← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1512955] [NEW] Race condition in nova/neutron when booting instance with xen driver

 

Public bug reported:

1. My environment is:
    Xenserver 6.5
    OpenStack latest master branch
    Neutron network with ML2 plugin, OVS driver

2. When I ran tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops it will always fail.
It failed at finding the new created instance's port before assign a floating ip to this port.

Relevant log file of tempest:
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
-------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 127, in wrapper
        return f(self, *func_args, **func_kwargs)
      File "tempest/scenario/test_network_basic_ops.py", line 398, in test_network_basic_ops
        self._setup_network_and_servers()
      File "tempest/scenario/test_network_basic_ops.py", line 123, in _setup_network_and_servers
        floating_ip = self.create_floating_ip(server)
      File "tempest/scenario/manager.py", line 774, in create_floating_ip
        port_id, ip4 = self._get_server_port_id_and_ip4(thing)
      File "tempest/scenario/manager.py", line 755, in _get_server_port_id_and_ip4
        % port_map)
      File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: 0 != 1: Found multiple IPv4 addresses: []. Unable to determine which port to target.
----------------------------------------------------------------------------------------------------------------------------------------------------

3. This is a failure that can be reproduced each time as long as we use
xen and neutron.

4. I tried investagating with this problem, it dues to nova/neutron race
condition when booting an instance under xen driver, since xen driver
doesn't handle neutron's "network-vif-plugged" notification event.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1512955

Title:
  Race condition in nova/neutron when booting instance with xen driver

Status in OpenStack Compute (nova):
  New

Bug description:
  1. My environment is:
      Xenserver 6.5
      OpenStack latest master branch
      Neutron network with ML2 plugin, OVS driver

  2. When I ran tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops it will always fail.
  It failed at finding the new created instance's port before assign a floating ip to this port.

  Relevant log file of tempest:
  tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
  -------------------------------------------------------------------------------------------------------------------------------------------------
  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "tempest/test.py", line 127, in wrapper
          return f(self, *func_args, **func_kwargs)
        File "tempest/scenario/test_network_basic_ops.py", line 398, in test_network_basic_ops
          self._setup_network_and_servers()
        File "tempest/scenario/test_network_basic_ops.py", line 123, in _setup_network_and_servers
          floating_ip = self.create_floating_ip(server)
        File "tempest/scenario/manager.py", line 774, in create_floating_ip
          port_id, ip4 = self._get_server_port_id_and_ip4(thing)
        File "tempest/scenario/manager.py", line 755, in _get_server_port_id_and_ip4
          % port_map)
        File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
          self.assertThat(observed, matcher, message)
        File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
          raise mismatch_error
      testtools.matchers._impl.MismatchError: 0 != 1: Found multiple IPv4 addresses: []. Unable to determine which port to target.
  ----------------------------------------------------------------------------------------------------------------------------------------------------

  3. This is a failure that can be reproduced each time as long as we
  use xen and neutron.

  4. I tried investagating with this problem, it dues to nova/neutron
  race condition when booting an instance under xen driver, since xen
  driver doesn't handle neutron's "network-vif-plugged" notification
  event.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1512955/+subscriptions


Follow ups