yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1946666] Re: [ovn] neutron_ovn_db_sync_util crashes (ACL already exists)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Daniel Speichert <1946666@xxxxxxxxxxxxxxxxxx>
Date: Fri, 11 Feb 2022 15:40:37 -0000
Reply-to: Bug 1946666 <1946666@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1946666

Title:
  [ovn] neutron_ovn_db_sync_util crashes (ACL already exists)

Status in neutron:
  Fix Released

Bug description:
  Doing fresh install on Xena, running neutron-ovn-db-sync-util in
  repair mode crashes on second run on an empty Neutron database - it
  works exactly once.

  The same issue occurs on version of Neutron from Wallaby.

  OVN version: 21.09.0~git20210922.6549e5842-0ubuntu1~cloud0

  # neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --nouse-journal
  11/Oct/2021:17:53:17.429 382043 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [-] Started Neutron OVN db sync
  11/Oct/2021:17:53:17.778 382043 INFO neutron.manager [-] Loading core plugin: neutron.cmd.ovn.neutron_ovn_db_sync_util.Ml2Plugin
  11/Oct/2021:17:53:17.780 382043 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'vlan']
  11/Oct/2021:17:53:17.781 382043 INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed
  11/Oct/2021:17:53:17.782 382043 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict([('physnet1', [(2200, 2300)])])
  11/Oct/2021:17:53:17.782 382043 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan']
  11/Oct/2021:17:53:17.783 382043 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'vlan'])
  11/Oct/2021:17:53:17.783 382043 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['vlan']
  11/Oct/2021:17:53:17.783 382043 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos']
  11/Oct/2021:17:53:17.784 382043 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos']
  11/Oct/2021:17:53:17.784 382043 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos']
  11/Oct/2021:17:53:17.784 382043 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn-sync']
  11/Oct/2021:17:53:17.784 382043 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn-sync']
  11/Oct/2021:17:53:17.784 382043 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn-sync']
  11/Oct/2021:17:53:17.785 382043 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling.
  11/Oct/2021:17:53:17.785 382043 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
  11/Oct/2021:17:53:17.785 382043 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
  11/Oct/2021:17:53:17.785 382043 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'
  11/Oct/2021:17:53:17.980 382043 INFO neutron.plugins.ml2.drivers.type_vlan [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] VlanTypeDriver initialization complete
  11/Oct/2021:17:53:17.980 382043 INFO neutron.plugins.ml2.managers [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Initializing extension driver 'port_security'
  11/Oct/2021:17:53:17.981 382043 INFO neutron.plugins.ml2.extensions.port_security [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] PortSecurityExtensionDriver initialization complete
  11/Oct/2021:17:53:17.981 382043 INFO neutron.plugins.ml2.managers [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Initializing extension driver 'qos'
  11/Oct/2021:17:53:17.981 382043 INFO neutron.plugins.ml2.managers [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Initializing mechanism driver 'ovn-sync'
  11/Oct/2021:17:53:17.981 382043 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Starting OVNMechanismDriver
  11/Oct/2021:17:53:17.999 382043 INFO neutron.services.logapi.drivers.ovn.driver [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] OVN logging driver registered
  11/Oct/2021:17:53:17.999 382043 INFO neutron.plugins.ml2.plugin [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Modular L2 Plugin initialization complete
  11/Oct/2021:17:53:18.000 382043 INFO neutron.plugins.ml2.managers [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Got port-security extension from driver 'port_security'
  11/Oct/2021:17:53:18.000 382043 INFO neutron.extensions.vlantransparent [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Disabled vlantransparent extension.
  11/Oct/2021:17:53:18.001 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: neutron.services.ovn_l3.plugin.OVNL3RouterPlugin
  11/Oct/2021:17:53:18.007 382043 INFO neutron.services.ovn_l3.plugin [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Starting OVNL3RouterPlugin
  11/Oct/2021:17:53:18.008 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: neutron.services.segments.plugin.Plugin
  11/Oct/2021:17:53:18.068 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: port_forwarding
  11/Oct/2021:17:53:18.071 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading service plugin neutron.services.ovn_l3.plugin.OVNL3RouterPlugin, it is required by port_forwarding
  11/Oct/2021:17:53:18.072 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: auto_allocate
  11/Oct/2021:17:53:18.073 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: tag
  11/Oct/2021:17:53:18.075 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: timestamp
  11/Oct/2021:17:53:18.077 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: network_ip_availability
  11/Oct/2021:17:53:18.079 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: flavors
  11/Oct/2021:17:53:18.080 382043 INFO neutron.manager [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Loading Plugin: revisions
  11/Oct/2021:17:53:18.082 382043 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Starting OVNClientQosExtension
  11/Oct/2021:17:53:18.082 382043 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Sync for Northbound db started with mode : repair
  11/Oct/2021:17:53:18.183 382043 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] ACLs-to-be-added 2 ACLs-to-be-removed 2
  11/Oct/2021:17:53:18.183 382043 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] ACL found in Neutron but not in OVN DB for port group neutron_pg_drop
  11/Oct/2021:17:53:18.184 382043 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] ACL found in Neutron but not in OVN DB for port group neutron_pg_drop
  11/Oct/2021:17:53:18.185 382043 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
      command.run_idl(txn)
    File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 123, in run_idl
      raise RuntimeError("ACL (%s, %s, %s) already exists" % (
  RuntimeError: ACL (from-lport, 1001, inport == @neutron_pg_drop && ip) already exists

  11/Oct/2021:17:53:18.186 382043 CRITICAL neutron_ovn_db_sync_util [req-c4c58a3a-cf52-49a4-84c8-ce8d5def2825 - - - - -] Unhandled error: RuntimeError: ACL (from-lport, 1001, inport == @neutron_pg_drop && ip) already exists
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     sys.exit(main())
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 235, in main
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 104, in do_sync
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     self.sync_acls(ctx)
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 302, in sync_acls
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     txn.add(self.ovn_api.pg_acl_add(**acla))
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     next(self.gen)
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 274, in transaction
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     yield t
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     next(self.gen)
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 110, in transaction
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     del self._nested_txns_map[cur_thread_id]
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 61, in __exit__
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     self.result = self.commit()
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     raise result.ex
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     txn.results.put(txn.do_commit())
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     command.run_idl(txn)
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 123, in run_idl
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util     raise RuntimeError("ACL (%s, %s, %s) already exists" % (
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util RuntimeError: ACL (from-lport, 1001, inport == @neutron_pg_drop && ip) already exists
  11/Oct/2021:17:53:18.186 382043 ERROR neutron_ovn_db_sync_util

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1946666/+subscriptions
References

[Bug 1946666] [NEW] [ovn] neutron_ovn_db_sync_util crashes (ACL already exists)
From: Daniel Speichert, 2021-10-11