yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2111272] [NEW] security group delete fails with PgDelCommand.*exceeded timeout 180 seconds, cause: Result queue is empty

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <2111272@xxxxxxxxxxxxxxxxxx>
Date: Mon, 19 May 2025 14:50:36 -0000
Reply-to: Bug 2111272 <2111272@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager [None req-b332b601-98dc-4f12-9235-6cf5943fb71e cinder_grenade cinder_grenade] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group-3711357 security_group, after_delete: ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55001	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55002	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 54, in commit
55003	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     result = self.results.get(timeout=self.timeout)
55004	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
55005	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     try:
55006	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
55007	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     try:
55008	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
55009	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     return self.greenlet.switch()
55010	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager _queue.Empty
55011	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 
55012	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager The above exception was the direct cause of the following exception:
55013	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 
55014	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
55015	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
55016	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     callback.method(resource, event, trigger, payload=payload)
55017	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 430, in _delete_security_group
55018	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group(context,
55019	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2583, in delete_security_group
55020	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with self._nb_idl.transaction(check_error=True) as txn:
55021	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55022	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     next(self.gen)
55023	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 274, in transaction
55024	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
55025	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
55026	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     next(self.gen)
55027	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
55028	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with self.create_transaction(
55029	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
55030	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     self.result = self.commit()
55031	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 56, in commit
55032	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     raise exceptions.TimeoutException(
55033	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
55034	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 

https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/screen-
neutron-api.txt

seen during grenade:

025-05-19 10:35:51.109 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:289 :   openstack security group delete cinder_grenade
14212	2025-05-19 10:40:52.922 | Failed to delete group with name or ID 'cinder_grenade': HttpException: 500: Server Error for url: http://200.225.47.46/networking/v2.0/security-groups/cf5c9692-6de1-4186-b64f-5317563d14da, 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: webmaster@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.: Apache/2.4.52 (Ubuntu) Server at 200.225.47.46 Port 80
14213	2025-05-19 10:40:52.922 | 1 of 1 groups failed to delete.

https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/grenade.sh_log.txt#14211-14213

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2111272

Title:
  security group delete fails with PgDelCommand.*exceeded timeout 180
  seconds, cause: Result queue is empty

Status in neutron:
  New

Bug description:
  May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager [None req-b332b601-98dc-4f12-9235-6cf5943fb71e cinder_grenade cinder_grenade] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group-3711357 security_group, after_delete: ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
  55001	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  55002	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 54, in commit
  55003	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     result = self.results.get(timeout=self.timeout)
  55004	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 320, in get
  55005	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     try:
  55006	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 139, in wait
  55007	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     try:
  55008	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
  55009	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     return self.greenlet.switch()
  55010	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager _queue.Empty
  55011	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 
  55012	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager The above exception was the direct cause of the following exception:
  55013	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 
  55014	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  55015	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 189, in _notify_loop
  55016	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     callback.method(resource, event, trigger, payload=payload)
  55017	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 430, in _delete_security_group
  55018	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group(context,
  55019	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2583, in delete_security_group
  55020	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with self._nb_idl.transaction(check_error=True) as txn:
  55021	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  55022	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     next(self.gen)
  55023	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/old/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 274, in transaction
  55024	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
  55025	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  55026	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     next(self.gen)
  55027	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
  55028	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     with self.create_transaction(
  55029	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
  55030	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     self.result = self.commit()
  55031	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/data/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 56, in commit
  55032	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager     raise exceptions.TimeoutException(
  55033	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager ovsdbapp.exceptions.TimeoutException: Commands [PgDelCommand(_result=None, name=pg_cf5c9692_6de1_4186_b64f_5317563d14da, if_exists=True)] exceeded timeout 180 seconds, cause: Result queue is empty
  55034	May 19 10:38:53.079114 np0040803311 devstack@neutron-api.service[75684]: ERROR neutron_lib.callbacks.manager 

  https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/screen-
  neutron-api.txt

  seen during grenade:

  025-05-19 10:35:51.109 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:289 :   openstack security group delete cinder_grenade
  14212	2025-05-19 10:40:52.922 | Failed to delete group with name or ID 'cinder_grenade': HttpException: 500: Server Error for url: http://200.225.47.46/networking/v2.0/security-groups/cf5c9692-6de1-4186-b64f-5317563d14da, 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: webmaster@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.: Apache/2.4.52 (Ubuntu) Server at 200.225.47.46 Port 80
  14213	2025-05-19 10:40:52.922 | 1 of 1 groups failed to delete.

  https://zuul.opendev.org/t/openstack/build/34e1c6cf11e04d17bbac42bf399d76b5/log/controller/logs/grenade.sh_log.txt#14211-14213

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2111272/+subscriptions