yahoo-eng-team team mailing list archive

[OpenStack Infra <1818613@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/641117
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92f1281b696c79133609d3c04b467ac7ea9f4337
Submitter: Zuul
Branch:    master

commit 92f1281b696c79133609d3c04b467ac7ea9f4337
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Tue Mar 5 18:37:44 2019 +0000

    Add a more robust method to check OVSDB values in BaseOVSTestCase
    
    Sometimes, when the OVSDB is too loaded (that could happen during the
    functional tests), there is a delay between the OVSDB post transaction
    end and when the register (new or updated) can be read. Although this is
    something that should not happen (considering the OVSDB is transactional),
    tests should deal with this inconvenience and provide a robust method to
    retrieve a value and at the same time check the value. This new method
    should provide a retrieving mechanism to read again the value in case of
    discordance.
    
    In order to solve the gate problem ASAP, another bug is fixed in this
    patch: to skip the QoS removal when OVS agent is initialized during
    funtional tests
    
    When executing functional tests, several OVS QoS policies specific for
    minimum bandwidth rules [1]. Because during the functional tests
    execution several threads can create more than one minimum bandwidth
    QoS policy (something in a production environment cannot happen), the
    OVS QoS driver must skip the execution of [2] to avoid removing other
    QoS created in parellel in other tests.
    
    This patch is marking as unstable "test_min_bw_qos_policy_rule_lifecycle"
    and "test_bw_limit_qos_port_removed". Those tests will be investigated
    once the CI gates are stable.
    
    [1] Those QoS policies are created only to hold minimum bandwidth rules.
        Those policies are marked with:
           external_ids: {'_type'='minimum_bandwidth'}
    [2] https://github.com/openstack/neutron/blob/d6fba30781c5f4e63beeda04d065226660fc92b6/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py#L43
    
    Closes-Bug: #1818613
    Closes-Bug: #1818859
    Related-Bug: #1819125
    
    Change-Id: Ia725cc1b36bc3630d2891f86f76b13c16f6cc37c


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1818613

Title:
  Functional qos related tests fails often

Status in neutron:
  Fix Released

Bug description:
  Various QoS related tests are failing often recently. In all cases
  reason is the same: "ovsdbapp.backend.ovs_idl.idlutils.RowNotFound:
  Cannot find Port with name=cc566ab0-4201-44b5-ae89-d342284ffdd6"
  during "_minimum_bandwidth_initialize".

  Stacktrace:

  ft1.1: neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_policy_rule_delete(ingress)_StringException: Traceback (most recent call last):
    File "neutron/tests/base.py", line 174, in func
      return f(self, *args, **kwargs)
    File "neutron/tests/functional/agent/l2/extensions/test_ovs_agent_qos_extension.py", line 354, in test_policy_rule_delete
      port_dict = self._create_port_with_qos()
    File "neutron/tests/functional/agent/l2/extensions/test_ovs_agent_qos_extension.py", line 172, in _create_port_with_qos
      self.setup_agent_and_ports([port_dict])
    File "neutron/tests/functional/agent/l2/base.py", line 375, in setup_agent_and_ports
      ancillary_bridge=ancillary_bridge)
    File "neutron/tests/functional/agent/l2/base.py", line 116, in create_agent
      ext_mgr, self.config)
    File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 256, in __init__
      self.connection, constants.EXTENSION_DRIVER_TYPE, agent_api)
    File "neutron/agent/agent_extensions_manager.py", line 54, in initialize
      extension.obj.initialize(connection, driver_type)
    File "neutron/agent/l2/extensions/qos.py", line 207, in initialize
      self.qos_driver.initialize()
    File "neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 57, in initialize
      self._minimum_bandwidth_initialize()
    File "neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 52, in _minimum_bandwidth_initialize
      self.br_int.clear_minimum_bandwidth_qos()
    File "neutron/agent/common/ovs_lib.py", line 1006, in clear_minimum_bandwidth_qos
      self.ovsdb.db_destroy('QoS', qos_id).execute(check_error=True)
    File "/opt/stack/new/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
      txn.add(self)
    File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
      self.gen.next()
    File "/opt/stack/new/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/ovsdbapp/api.py", line 112, in transaction
      del self._nested_txns_map[cur_thread_id]
    File "/opt/stack/new/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/ovsdbapp/api.py", line 69, in __exit__
      self.result = self.commit()
    File "/opt/stack/new/neutron/.tox/dsvm-functional-python27/local/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
      raise result.ex
  ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port with name=cc566ab0-4201-44b5-ae89-d342284ffdd6

  Example failure: http://logs.openstack.org/74/640874/1/check/neutron-
  functional-python27/d51cd50/logs/testr_results.html.gz

  Logstash query:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2052%2C%20in%20_minimum_bandwidth_initialize%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1818613/+subscriptions