yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95890
[Bug 2111272] [NEW] security group delete fails with PgDelCommand.*exceeded timeout 180 seconds, cause: Result queue is empty
Public bug reported:
May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager [None req-b332b601-98dc-4f12-9235-6cf5943fb71e cinder_grenade cinder_grenade] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group-3711357 security_group, after_delete: ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55001 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55002 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 54, in commit
55003 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager result = self.results.get(timeout=self.timeout)
55004 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
55005 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager try:
55006 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
55007 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager try:
55008 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
55009 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager return self.greenlet.switch()
55010 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager _queue.Empty
55011 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
55012 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager The above exception was the direct cause of the following exception:
55013 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
55014 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55015 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
55016 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager callback.method(resource, event, trigger, payload=payload)
55017 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 430, in _delete_security_group
55018 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager self._ovn_client.delete_security_group(context,
55019 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2583, in delete_security_group
55020 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn:
55021 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55022 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager next(self.gen)
55023 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 274, in transaction
55024 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
55025 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55026 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager next(self.gen)
55027 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
55028 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with self.create_transaction(
55029 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
55030 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager self.result = self.commit()
55031 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 56, in commit
55032 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager raise exceptions.TimeoutException(
55033 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55034 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/screen-
neutron-api.txt
seen during grenade:
025-05-19 10:35:51.109 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:289 : openstack security group delete cinder_grenade
14212 2025-05-19 10:40:52.922 | Failed to delete group with name or ID 'cinder_grenade': HttpException: 500: Server Error for url: http://200.225.47.46/networking/v2.0/security-groups/cf5c9692-6de1-4186-b64f-5317563d14da, 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: webmaster@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.: Apache/2.4.52 (Ubuntu) Server at 200.225.47.46 Port 80
14213 2025-05-19 10:40:52.922 | 1 of 1 groups failed to delete.
https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/grenade.sh_log.txt#14211-14213
** Affects: neutron
Importance: Undecided
Status: New
** Tags: gate-failure
** Tags added: gate-failure
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2111272
Title:
security group delete fails with PgDelCommand.*exceeded timeout 180
seconds, cause: Result queue is empty
Status in neutron:
New
Bug description:
May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager [None req-b332b601-98dc-4f12-9235-6cf5943fb71e cinder_grenade cinder_grenade] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group-3711357 security_group, after_delete: ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55001 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55002 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 54, in commit
55003 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager result = self.results.get(timeout=self.timeout)
55004 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
55005 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager try:
55006 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
55007 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager try:
55008 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
55009 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager return self.greenlet.switch()
55010 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager _queue.Empty
55011 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
55012 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager The above exception was the direct cause of the following exception:
55013 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
55014 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55015 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
55016 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager callback.method(resource, event, trigger, payload=payload)
55017 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 430, in _delete_security_group
55018 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager self._ovn_client.delete_security_group(context,
55019 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2583, in delete_security_group
55020 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn:
55021 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55022 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager next(self.gen)
55023 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 274, in transaction
55024 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
55025 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55026 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager next(self.gen)
55027 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
55028 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager with self.create_transaction(
55029 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
55030 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager self.result = self.commit()
55031 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 56, in commit
55032 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager raise exceptions.TimeoutException(
55033 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55034 May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager
https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/screen-
neutron-api.txt
seen during grenade:
025-05-19 10:35:51.109 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:289 : openstack security group delete cinder_grenade
14212 2025-05-19 10:40:52.922 | Failed to delete group with name or ID 'cinder_grenade': HttpException: 500: Server Error for url: http://200.225.47.46/networking/v2.0/security-groups/cf5c9692-6de1-4186-b64f-5317563d14da, 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: webmaster@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.: Apache/2.4.52 (Ubuntu) Server at 200.225.47.46 Port 80
14213 2025-05-19 10:40:52.922 | 1 of 1 groups failed to delete.
https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/grenade.sh_log.txt#14211-14213
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2111272/+subscriptions