yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1845176] Re: Removing of QoS queue in neutron-ovs-agent fails due to existing references

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1845176@xxxxxxxxxxxxxxxxxx>
Date: Wed, 22 Jan 2020 05:36:34 -0000
Reply-to: Bug 1845176 <1845176@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/687922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=83d7eb961dc8a5d4e815bc3f110f8523a532caeb
Submitter: Zuul
Branch:    master

commit 83d7eb961dc8a5d4e815bc3f110f8523a532caeb
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Tue Oct 15 15:51:40 2019 +0000

    Add OVS QoS driver cache for minimum bandwidth rules
    
    1. Add OVS QoS driver cache
    Added minimum bandwidth rules cache in the OVS QoS driver. If a new
    port is detected in the integration bridge, managed by the OVS agent
    driver, the QoS extension is applied (if configured).
    
    If this new port does not have a QoS rule, by default the QoS
    configuration is reset in this port. Because this port has no Queue
    related, the call "QosOVSAgentDriver.delete_minimum_bandwidth" should
    not execute the rule cleanup.
    
    This QoS rule cache implementation per porti is currently used in the
    OVS QoS driver for the DSCP and the bandwitdh limit rules.
    
    2. If a Queue cannot be deleted, log the QoS registers using it
    If a Queue is still used in a QoS register in the OVSDB, it cannot
    be deleted. With the current design, only one QoS rule is created in
    the database and several Queues are assigned to it.
    
    If something external to the OVS agent is handling the Queues and by
    mistake a Queue is assigned to other QoS registers, this error is now
    logged before raising the exception.
    
    Closes-Bug: #1845176
    Change-Id: Ia9077fc20e4ca360819a2e368c8c1f9250e5a6d8


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1845176

Title:
  Removing of QoS queue in neutron-ovs-agent fails due to existing
  references

Status in neutron:
  Fix Released

Bug description:
  It happend in fullstack test:
  neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress).
  Test failed and in neutron-openvswitch-agent logs we found error like:

  2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432
  2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103
  2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     yield self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     t.add(self)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     next(self.gen)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise result.ex
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     txn.results.put(txn.do_commit())
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise RuntimeError(msg)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     yield self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed))
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ext_manager.delete_port(self.context, {'port_id': device})
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     extension.obj.delete_port(context, data)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._process_reset_port(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.qos_driver.delete(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._handle_rule_delete(port, rule_type)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     handler(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.br_int.delete_minimum_bandwidth_queue(port['port_id'])
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._delete_queue(queue['_uuid'])
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     t.add(self)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     next(self.gen)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     del self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.result = self.commit()
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise result.ex
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     txn.results.put(txn.do_commit())
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(msg)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

  
  Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1845176/+subscriptions
References

[Bug 1845176] [NEW] Removing of QoS queue in neutron-ovs-agent fails due to existing references
From: Slawek Kaplonski, 2019-09-24