yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2116337] [NEW] master networking sfc jobs broken as ovs agent crashing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2116337@xxxxxxxxxxxxxxxxxx>
Date: Thu, 10 Jul 2025 15:43:49 -0000
Reply-to: Bug 2116337 <2116337@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Example failure:-
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b13/openstack/b13fbba36a314fd49a95ddcc07dbbd00/testr_results.html

OVS agent crashes as:-
Jul 01 09:59:33.124631 npc1563e6e31d04 neutron-openvswitch-agent[63989]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Bridge, record=br-int, col_values=(('other_config', {'mcast-snooping-disable-flood-unregistered': 'true'}),), if_exists=True) {{(pid=63989) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}}
Jul 01 09:59:33.127156 npc1563e6e31d04 neutron-openvswitch-agent[63989]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000e29279f4354f
Jul 01 09:59:33.129617 npc1563e6e31d04 neutron-openvswitch-agent[63989]: Traceback (most recent call last):
Jul 01 09:59:33.129967 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
Jul 01 09:59:33.129967 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     timer()
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     cb(*args, **kw)
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     waiter.switch()
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]: greenlet.error: Cannot switch to a different thread
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Current:  <greenlet.greenlet object at 0x7ec2c7e848c0 (otid=0x7ec2e5a08f00) suspended active started main>
Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Expected: <greenlet.greenlet object at 0x7ec2df5ad8c0 (otid=0x7ec2df56aa30) suspended active started main>
Jul 01 09:59:56.211137 npc1563e6e31d04 neutron-openvswitch-agent[63989]: Traceback (most recent call last):
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     timer()
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     cb(*args, **kw)
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     waiter.switch()
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]: greenlet.error: Cannot switch to a different thread
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Current:  <greenlet.greenlet object at 0x7ec2df5e4b80 (otid=0x7ec2c7f94cf0) suspended active started main>
Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Expected: <greenlet.greenlet object at 0x7ec2c7e848c0 (otid=0x7ec2e5a08f00) suspended active started main>


Builds:- https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-sfc&job_name=networking-sfc-tempest&branch=master&skip=0

Issue triggered by patch
https://review.opendev.org/c/openstack/neutron/+/937765

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2116337

Title:
  master networking sfc jobs broken as ovs agent crashing

Status in neutron:
  New

Bug description:
  Example failure:-
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b13/openstack/b13fbba36a314fd49a95ddcc07dbbd00/testr_results.html

  OVS agent crashes as:-
  Jul 01 09:59:33.124631 npc1563e6e31d04 neutron-openvswitch-agent[63989]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Bridge, record=br-int, col_values=(('other_config', {'mcast-snooping-disable-flood-unregistered': 'true'}),), if_exists=True) {{(pid=63989) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}}
  Jul 01 09:59:33.127156 npc1563e6e31d04 neutron-openvswitch-agent[63989]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000e29279f4354f
  Jul 01 09:59:33.129617 npc1563e6e31d04 neutron-openvswitch-agent[63989]: Traceback (most recent call last):
  Jul 01 09:59:33.129967 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
  Jul 01 09:59:33.129967 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     timer()
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     cb(*args, **kw)
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     waiter.switch()
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]: greenlet.error: Cannot switch to a different thread
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Current:  <greenlet.greenlet object at 0x7ec2c7e848c0 (otid=0x7ec2e5a08f00) suspended active started main>
  Jul 01 09:59:33.130180 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Expected: <greenlet.greenlet object at 0x7ec2df5ad8c0 (otid=0x7ec2df56aa30) suspended active started main>
  Jul 01 09:59:56.211137 npc1563e6e31d04 neutron-openvswitch-agent[63989]: Traceback (most recent call last):
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     timer()
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     cb(*args, **kw)
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:     waiter.switch()
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]: greenlet.error: Cannot switch to a different thread
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Current:  <greenlet.greenlet object at 0x7ec2df5e4b80 (otid=0x7ec2c7f94cf0) suspended active started main>
  Jul 01 09:59:56.211595 npc1563e6e31d04 neutron-openvswitch-agent[63989]:         Expected: <greenlet.greenlet object at 0x7ec2c7e848c0 (otid=0x7ec2e5a08f00) suspended active started main>

  
  Builds:- https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-sfc&job_name=networking-sfc-tempest&branch=master&skip=0

  Issue triggered by patch
  https://review.opendev.org/c/openstack/neutron/+/937765

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2116337/+subscriptions
Follow ups

[Bug 2116337] Re: master networking sfc jobs broken as ovs agent crashing
From: Rodolfo Alonso, 2025-07-14
[Bug 2116337] Re: master networking sfc jobs broken as ovs agent crashing
From: yatin, 2025-07-10