← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #92644

[Bug 2027931] [NEW] Neutron Sync Repair Failed

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

i have 3 controller openstack, but the network agent often 504 Gateway timeout. i checked the sync status of neutron `neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
`. the logs looks like this :

2023-07-11 09:57:16.905 31504 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Sync for Northbound db started with mode : log
2023-07-11 09:57:45.263 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 found in Neutron but not in OVN
2023-07-11 09:58:05.031 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Port found in OVN but not in Neutron, port_id=9576db78-8fa9-462a-9b15-edcc07af31bc
2023-07-11 09:58:05.031 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Port found in OVN but not in Neutron, port_id=8c5d7903-22f2-4a42-9dd7-53aa8018ab76
2023-07-11 10:00:07.003 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Router found in Neutron but not in OVN DB, router id=012c65cd-ee98-4640-b515-f89abdf7639f   

So i want to repair the neutron `/var/lib/kolla/venv/bin/neutron-ovn-db-
sync-util --config-file /etc/neutron/neutron.conf --config-file
/etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair`.
But the repair failed because timeout.

2023-07-14 19:51:09.322 65827 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Starting OVNClientQosExtension
2023-07-14 19:51:09.323 65827 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Sync for Northbound db started with mode : repair
2023-07-14 19:51:55.957 65827 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 found in Neutron but not in OVN
2023-07-14 19:51:55.957 65827 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Creating metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 in OVN
2023-07-14 19:56:19.289 65827 CRITICAL neutron_ovn_db_sync_util [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Unhandled error: ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.AddLSwitchPortCommand object at 0x7fed05ade9b0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec7f0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec828>] exceeded timeout 180 seconds
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 55, in commit
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     result = self.results.get(timeout=self.timeout)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/queue.py", line 172, in get
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     raise Empty
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util queue.Empty
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/bin/neutron-ovn-db-sync-util", line 8, in <module>
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     sys.exit(main())
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 869, in sync_networks_ports_and_dhcp_opts
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._sync_metadata_ports(ctx, db_ports)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 789, in _sync_metadata_ports
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._create_port_in_ovn(ctx, port)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 111, in _create_port_in_ovn
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._ovn_client.create_port(ctx, port)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 440, in create_port
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._qos_driver.create_port(txn, port, port_type=port_info.type)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     next(self.gen)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 240, in transaction
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     yield t
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     next(self.gen)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     del self._nested_txns_map[cur_thread_id]
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self.result = self.commit()
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 58, in commit
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     timeout=self.timeout)
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.AddLSwitchPortCommand object at 0x7fed05ade9b0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec7f0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec828>] exceeded timeout 180 seconds
2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 

I have tried to increase the timeout to 180s, but no luck.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: kolla-ansible neutron openstack

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2027931

Title:
  Neutron Sync Repair Failed

Status in neutron:
  New

Bug description:
  i have 3 controller openstack, but the network agent often 504 Gateway timeout. i checked the sync status of neutron `neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
  `. the logs looks like this :

  2023-07-11 09:57:16.905 31504 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Sync for Northbound db started with mode : log
  2023-07-11 09:57:45.263 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 found in Neutron but not in OVN
  2023-07-11 09:58:05.031 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Port found in OVN but not in Neutron, port_id=9576db78-8fa9-462a-9b15-edcc07af31bc
  2023-07-11 09:58:05.031 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Port found in OVN but not in Neutron, port_id=8c5d7903-22f2-4a42-9dd7-53aa8018ab76
  2023-07-11 10:00:07.003 31504 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-077b1123-58ac-424b-90bd-9af09809d51e - - - - -] Router found in Neutron but not in OVN DB, router id=012c65cd-ee98-4640-b515-f89abdf7639f   

  So i want to repair the neutron `/var/lib/kolla/venv/bin/neutron-ovn-
  db-sync-util --config-file /etc/neutron/neutron.conf --config-file
  /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair`.
  But the repair failed because timeout.

  2023-07-14 19:51:09.322 65827 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Starting OVNClientQosExtension
  2023-07-14 19:51:09.323 65827 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Sync for Northbound db started with mode : repair
  2023-07-14 19:51:55.957 65827 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 found in Neutron but not in OVN
  2023-07-14 19:51:55.957 65827 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Creating metadata port 59e422ea-dbcd-40c7-a458-235f42a9d8ad for network a111b028-569e-4ba4-9448-5f4c09685221 in OVN
  2023-07-14 19:56:19.289 65827 CRITICAL neutron_ovn_db_sync_util [req-04c202a4-3383-49eb-b9db-37ad0008da5e - - - - -] Unhandled error: ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.AddLSwitchPortCommand object at 0x7fed05ade9b0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec7f0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec828>] exceeded timeout 180 seconds
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 55, in commit
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     result = self.results.get(timeout=self.timeout)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/queue.py", line 172, in get
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     raise Empty
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util queue.Empty
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/bin/neutron-ovn-db-sync-util", line 8, in <module>
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     sys.exit(main())
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 869, in sync_networks_ports_and_dhcp_opts
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._sync_metadata_ports(ctx, db_ports)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 789, in _sync_metadata_ports
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._create_port_in_ovn(ctx, port)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 111, in _create_port_in_ovn
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._ovn_client.create_port(ctx, port)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 440, in create_port
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self._qos_driver.create_port(txn, port, port_type=port_info.type)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     next(self.gen)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 240, in transaction
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     yield t
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     next(self.gen)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     del self._nested_txns_map[cur_thread_id]
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     self.result = self.commit()
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 58, in commit
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util     timeout=self.timeout)
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.AddLSwitchPortCommand object at 0x7fed05ade9b0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec7f0>, <ovsdbapp.schema.ovn_northbound.commands.QoSDelCommand object at 0x7fed05aec828>] exceeded timeout 180 seconds
  2023-07-14 19:56:19.289 65827 ERROR neutron_ovn_db_sync_util 

  I have tried to increase the timeout to 180s, but no luck.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2027931/+subscriptions
  Thread Previous • Date Previous • • Thread Next