yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1845176] [NEW] Removing of QoS queue in neutron-ovs-agent fails due to existing references

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1845176@xxxxxxxxxxxxxxxxxx>
Date: Tue, 24 Sep 2019 09:57:22 -0000
Reply-to: Bug 1845176 <1845176@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It happend in fullstack test:
neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress).
Test failed and in neutron-openvswitch-agent logs we found error like:

2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432
2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103
2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     t.add(self)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     next(self.gen)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise result.ex
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     txn.results.put(txn.do_commit())
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise RuntimeError(msg)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed))
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ext_manager.delete_port(self.context, {'port_id': device})
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     extension.obj.delete_port(context, data)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._process_reset_port(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.qos_driver.delete(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._handle_rule_delete(port, rule_type)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     handler(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.br_int.delete_minimum_bandwidth_queue(port['port_id'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._delete_queue(queue['_uuid'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     t.add(self)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     next(self.gen)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.result = self.commit()
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise result.ex
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     txn.results.put(txn.do_commit())
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(msg)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}


Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz

** Affects: neutron
     Importance: Medium
         Status: Confirmed


** Tags: gate-failure qos

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1845176

Title:
  Removing of QoS queue in neutron-ovs-agent fails due to existing
  references

Status in neutron:
  Confirmed

Bug description:
  It happend in fullstack test:
  neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress).
  Test failed and in neutron-openvswitch-agent logs we found error like:

  2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432
  2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103
  2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
  2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
      txn.results.put(txn.do_commit())
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
      raise RuntimeError(msg)
  RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     yield self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     t.add(self)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     next(self.gen)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise result.ex
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     txn.results.put(txn.do_commit())
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command     raise RuntimeError(msg)
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     yield self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed))
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ext_manager.delete_port(self.context, {'port_id': device})
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     extension.obj.delete_port(context, data)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._process_reset_port(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.qos_driver.delete(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._handle_rule_delete(port, rule_type)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     handler(port)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.br_int.delete_minimum_bandwidth_queue(port['port_id'])
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._delete_queue(queue['_uuid'])
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     t.add(self)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     next(self.gen)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     del self._nested_txns_map[cur_thread_id]
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.result = self.commit()
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise result.ex
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     txn.results.put(txn.do_commit())
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(msg)
  2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

  
  Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1845176/+subscriptions
Follow ups

[Bug 1845176] Re: Removing of QoS queue in neutron-ovs-agent fails due to existing references
From: OpenStack Infra, 2020-01-22