yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1802640] [NEW] TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt gate failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Sat, 10 Nov 2018 12:41:56 -0000
Reply-to: Bug 1802640 <1802640@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen here:

http://logs.openstack.org/40/617040/1/check/nova-
next/0a82b26/logs/screen-q-agt.txt.gz?level=TRACE

Nov 10 03:51:03.120446 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
Nov 10 03:51:03.120918 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
Nov 10 03:51:03.121258 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
Nov 10 03:51:03.121528 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     txn.add(self)
Nov 10 03:51:03.121801 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
Nov 10 03:51:03.122032 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.gen.next()
Nov 10 03:51:03.122299 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 112, in transaction
Nov 10 03:51:03.122563 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
Nov 10 03:51:03.122747 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 69, in __exit__
Nov 10 03:51:03.123003 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
Nov 10 03:51:03.123198 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 57, in commit
Nov 10 03:51:03.123418 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     timeout=self.timeout)
Nov 10 03:51:03.123647 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
Nov 10 03:51:03.123865 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command 
Nov 10 03:51:03.131466 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds Agent terminated!: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
Nov 10 03:51:03.134192 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl [-] Post-commit checks failed: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
Nov 10 03:51:03.134358 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl Traceback (most recent call last):
Nov 10 03:51:03.134471 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 40, in post_commit
Nov 10 03:51:03.134590 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     self.do_post_commit(txn)
Nov 10 03:51:03.134706 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 60, in do_post_commit
Nov 10 03:51:03.134821 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     timeout=self.timeout)
Nov 10 03:51:03.134989 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
Nov 10 03:51:03.135117 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl 
Nov 10 03:51:03.518750 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Main process exited, code=exited, status=1/FAILURE
Nov 10 03:51:03.519100 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Unit entered failed state.
Nov 10 03:51:03.519135 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Failed with result 'exit-code'.

This could just be an intermittent failure on slow CI nodes.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TimeoutException%3A%20Commands%20%5B%3Covsdbapp.schema.open_vswitch.commands.SetFailModeCommand%20object%20at%5C%22%20AND%20message%3A%5C%22exceeded%20timeout%2010%20seconds%5C%22%20AND%20tags%3A%5C%22screen-q-agt.txt%5C%22&from=10d

Reporting this for tracking in elastic-recheck.

** Affects: neutron
     Importance: Undecided
         Status: Confirmed


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1802640

Title:
  TimeoutException: Commands
  [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt
  gate failure

Status in neutron:
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/40/617040/1/check/nova-
  next/0a82b26/logs/screen-q-agt.txt.gz?level=TRACE

  Nov 10 03:51:03.120446 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.120918 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  Nov 10 03:51:03.121258 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
  Nov 10 03:51:03.121528 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     txn.add(self)
  Nov 10 03:51:03.121801 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  Nov 10 03:51:03.122032 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.gen.next()
  Nov 10 03:51:03.122299 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 112, in transaction
  Nov 10 03:51:03.122563 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     del self._nested_txns_map[cur_thread_id]
  Nov 10 03:51:03.122747 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/api.py", line 69, in __exit__
  Nov 10 03:51:03.123003 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     self.result = self.commit()
  Nov 10 03:51:03.123198 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 57, in commit
  Nov 10 03:51:03.123418 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command     timeout=self.timeout)
  Nov 10 03:51:03.123647 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.123865 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.backend.ovs_idl.command 
  Nov 10 03:51:03.131466 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds Agent terminated!: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.134192 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl [-] Post-commit checks failed: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.134358 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl Traceback (most recent call last):
  Nov 10 03:51:03.134471 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 40, in post_commit
  Nov 10 03:51:03.134590 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     self.do_post_commit(txn)
  Nov 10 03:51:03.134706 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl   File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py", line 60, in do_post_commit
  Nov 10 03:51:03.134821 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl     timeout=self.timeout)
  Nov 10 03:51:03.134989 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand object at 0x7fb4e1a94190>] exceeded timeout 10 seconds
  Nov 10 03:51:03.135117 ubuntu-xenial-ovh-bhs1-0000461070 neutron-openvswitch-agent[26802]: ERROR ovsdbapp.schema.open_vswitch.impl_idl 
  Nov 10 03:51:03.518750 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Main process exited, code=exited, status=1/FAILURE
  Nov 10 03:51:03.519100 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Unit entered failed state.
  Nov 10 03:51:03.519135 ubuntu-xenial-ovh-bhs1-0000461070 systemd[1]: devstack@q-agt.service: Failed with result 'exit-code'.

  This could just be an intermittent failure on slow CI nodes.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TimeoutException%3A%20Commands%20%5B%3Covsdbapp.schema.open_vswitch.commands.SetFailModeCommand%20object%20at%5C%22%20AND%20message%3A%5C%22exceeded%20timeout%2010%20seconds%5C%22%20AND%20tags%3A%5C%22screen-q-agt.txt%5C%22&from=10d

  Reporting this for tracking in elastic-recheck.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1802640/+subscriptions
Follow ups

[Bug 1802640] Re: TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt gate failure
From: Slawek Kaplonski, 2021-12-10