yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #81065
[Bug 1857016] Re: Possible double row.delete() call in ACL code
** Also affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1857016
Title:
Possible double row.delete() call in ACL code
Status in networking-ovn:
New
Status in neutron:
New
Bug description:
In the field, we've seen:
2019-11-11 14:10:04.765 54 ERROR ovsdbapp.backend.ovs_idl.transaction [req-bb3c2ce5-6a46-47d0-b544-772efa71158f - - - - -] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run
txn.results.put(txn.do_commit())
File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
command.run_idl(txn)
File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/commands.py", line 616, in run_idl
acl_del_obj.delete()
File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 970, in delete
assert self._changes is not None
AssertionError
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance [req-bb3c2ce5-6a46-47d0-b544-772efa71158f - - - - -] Failed to fix resource ebc7e039-81af-4f18-babf-8750fe24d5f0 (type: ports): AssertionError
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/networking_ovn/common/maintenance.py", line 232, in check_for_inconsistencies
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance self._fix_create_update(row)
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/networking_ovn/common/maintenance.py", line 178, in _fix_create_update
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance res_map['ovn_update'](n_obj)
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 495, in update_port
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance self.add_txns_to_remove_port_dns_records(txn, port_object)
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance self.gen.next()
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 139, in transaction
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance yield t
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance self.gen.next()
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 102, in transaction
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance del self._nested_txns_map[cur_thread_id]
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 59, in __exit__
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance self.result = self.commit()
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance raise result.ex
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance AssertionError
2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance
A cursory look at the python-ovs code makes it look like the
maintenance thread might be trying to delete the same row twice since
the only way Row._changes = None is in delete() already.
The ACL code passes around dicts which, being unhashable, can't be
added to sets to ensure uniqueness. In addition, from a db-schema
perspective the same ACL could be referenced from multiple objects.
Ultimately this code should be refactored, but a simple workaround for
now would be to do a try/except AssertionError around the row.delete()
since ignoring a 2nd attempted delete of the same row in a transaction
is safe.
To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-ovn/+bug/1857016/+subscriptions