yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1802640] Re: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt gate failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1802640@xxxxxxxxxxxxxxxxxx>
Date: Fri, 10 Dec 2021 14:46:22 -0000
Reply-to: Bug 1802640 <1802640@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
I don't think we still have such issues in the tempest jobs. It can
happen from time to time in the functional job maybe, but it's very rare
recently and IMHO this may be related to overloaded nodes simply. So I'm
closing that bug for now.

** Changed in: neutron
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1802640

Title:
  TimeoutException: Commands
  [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt
  gate failure

Status in neutron:
  Fix Released

Bug description:
  Seen here:

  http://logs.openstack.org/40/617040/1/check/nova-
  next/0a82b26/logs/screen-q-agt.txt.gz?level=TRACE

  Nov 10 03:51:03.120446 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.120918 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  Nov 10 03:51:03.121258 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
  Nov 10 03:51:03.121528 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     txn.add(self)
  Nov 10 03:51:03.121801 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  Nov 10 03:51:03.122032 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.gen.next()
  Nov 10 03:51:03.122299 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 112, in transaction
  Nov 10 03:51:03.122563 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
  Nov 10 03:51:03.122747 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 69, in __exit__
  Nov 10 03:51:03.123003 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
  Nov 10 03:51:03.123198 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 57, in commit
  Nov 10 03:51:03.123418 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     timeout=self.timeout)
  Nov 10 03:51:03.123647 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.123865 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command 
  Nov 10 03:51:03.131466 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds Agent terminated!: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.134192 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl [-] Post-commit checks failed: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.134358 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl Traceback (most recent call last):
  Nov 10 03:51:03.134471 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 40, in post_commit
  Nov 10 03:51:03.134590 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     self.do_post_commit(txn)
  Nov 10 03:51:03.134706 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 60, in do_post_commit
  Nov 10 03:51:03.134821 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     timeout=self.timeout)
  Nov 10 03:51:03.134989 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.135117 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl 
  Nov 10 03:51:03.518750 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Main process exited, code=exited, status=1/FAILURE
  Nov 10 03:51:03.519100 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Unit entered failed state.
  Nov 10 03:51:03.519135 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Failed with result 'exit-code'.

  This could just be an intermittent failure on slow CI nodes.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TimeoutException%3A%20Commands%20%5B%3Covsdbapp.schema.open_vswitch.commands.SetFailModeCommand%20object%20at%5C%22%20AND%20message%3A%5C%22exceeded%20timeout%2010%20seconds%5C%22%20AND%20tags%3A%5C%22screen-
  q-agt.txt%5C%22&from=10d

  Reporting this for tracking in elastic-recheck.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1802640/+subscriptions
References

[Bug 1802640] [NEW] TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt gate failure
From: Matt Riedemann, 2018-11-10