yahoo-eng-team team mailing list archive

[Brian Haley <1857016@xxxxxxxxxxxxxxxxxx>]

The UpdateACLsCommand class, where the code in question was, has been
completely removed in commit e748f3f2d800de6c84b6fe835edfa1385bc223b1 so
we can close this bug.

** Changed in: neutron
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1857016

Title:
  Possible double row.delete() call in ACL code

Status in networking-ovn:
  New
Status in neutron:
  Fix Released

Bug description:
  In the field, we've seen:

  2019-11-11 14:10:04.765 54 ERROR ovsdbapp.backend.ovs_idl.transaction [req-bb3c2ce5-6a46-47d0-b544-772efa71158f - - - - -] Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 99, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
      command.run_idl(txn)
    File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/commands.py", line 616, in run_idl
      acl_del_obj.delete()
    File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 970, in delete
      assert self._changes is not None
  AssertionError

  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance [req-bb3c2ce5-6a46-47d0-b544-772efa71158f - - - - -] Failed to fix resource ebc7e039-81af-4f18-babf-8750fe24d5f0 (type: ports): AssertionError
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/networking_ovn/common/maintenance.py", line 232, in check_for_inconsistencies
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     self._fix_create_update(row)
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/networking_ovn/common/maintenance.py", line 178, in _fix_create_update
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     res_map['ovn_update'](n_obj)
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 495, in update_port
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     self.add_txns_to_remove_port_dns_records(txn, port_object)
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     self.gen.next()
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 139, in transaction
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     yield t
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     self.gen.next()
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 102, in transaction
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     del self._nested_txns_map[cur_thread_id]
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 59, in __exit__
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     self.result = self.commit()
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance     raise result.ex
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance AssertionError
  2019-11-11 14:10:04.767 54 ERROR networking_ovn.common.maintenance

  A cursory look at the python-ovs code makes it look like the
  maintenance thread might be trying to delete the same row twice since
  the only way Row._changes = None is in delete() already.

  The ACL code passes around dicts which, being unhashable, can't be
  added to sets to ensure uniqueness. In addition, from a db-schema
  perspective the same ACL could be referenced from multiple objects.
  Ultimately this code should be refactored, but a simple workaround for
  now would be to do a try/except AssertionError around the row.delete()
  since ignoring a 2nd attempted delete of the same row in a transaction
  is safe.

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-ovn/+bug/1857016/+subscriptions