← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #18367

[Bug 1350903] [NEW] Ovs agent fails to kill ovsdb monitor properly

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The following log is observed in one of the deployments:

2014-07-31 14:42:58.744 28084 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Agent caught SIGTERM, quitting daemon loop. _handle_sigterm /usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1366
2014-07-31 14:42:58.748 28084 ERROR neutron.agent.linux.ovsdb_monitor [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Error received from ovsdb monitor: 2014-07-31T14:42:58Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
2014-07-31 14:42:58.760 28084 DEBUG neutron.agent.linux.async_process [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Halting async process [['ovsdb-client', 'monitor', 'Interface', 'name,ofport', '--format=json']] in response to an error. _handle_process_error /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:173
2014-07-31 14:42:58.763 28084 DEBUG neutron.agent.linux.utils [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Running command: ['ps', '--ppid', '28160', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:48
2014-07-31 14:42:58.880 28084 DEBUG neutron.agent.linux.utils [req-0698c817-970d-4c56-bd72-90fb37f1a134 None]
Command: ['ps', '--ppid', '28160', '-o', 'pid=']
Exit code: 1
Stdout: ''
Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:74
2014-07-31 14:43:00.040 28084 DEBUG neutron.agent.linux.async_process [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Halting async process [['ovsdb-client', 'monitor', 'Interface', 'name,ofport', '--format=json']]. stop /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:90
2014-07-31 14:43:00.052 28084 CRITICAL neutron [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] AssertionError: Trying to re-send() an already-triggered event.
2014-07-31 14:43:00.052 28084 TRACE neutron Traceback (most recent call last):
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2014-07-31 14:43:00.052 28084 TRACE neutron     sys.exit(main())
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1435, in main
2014-07-31 14:43:00.052 28084 TRACE neutron     agent.daemon_loop()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1363, in daemon_loop
2014-07-31 14:43:00.052 28084 TRACE neutron     self.rpc_loop(polling_manager=pm)
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2014-07-31 14:43:00.052 28084 TRACE neutron     self.gen.next()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 41, in get_polling_manager
2014-07-31 14:43:00.052 28084 TRACE neutron     pm.stop()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 108, in stop
2014-07-31 14:43:00.052 28084 TRACE neutron     self._monitor.stop()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 91, in stop
2014-07-31 14:43:00.052 28084 TRACE neutron     self._kill()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovsdb_monitor.py", line 108, in _kill
2014-07-31 14:43:00.052 28084 TRACE neutron     super(SimpleInterfaceMonitor, self)._kill(*args, **kwargs)
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 118, in _kill
2014-07-31 14:43:00.052 28084 TRACE neutron     self._kill_event.send()
2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 150, in send
2014-07-31 14:43:00.052 28084 TRACE neutron     assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2014-07-31 14:43:00.052 28084 TRACE neutron AssertionError: Trying to re-send() an already-triggered event.
2014-07-31 14:43:00.052 28084 TRACE neutron

** Affects: neutron
     Importance: Medium
     Assignee: Eugene Nikanorov (enikanorov)
         Status: New


** Tags: ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1350903

Title:
  Ovs agent fails to kill ovsdb monitor properly

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  The following log is observed in one of the deployments:

  2014-07-31 14:42:58.744 28084 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Agent caught SIGTERM, quitting daemon loop. _handle_sigterm /usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1366
  2014-07-31 14:42:58.748 28084 ERROR neutron.agent.linux.ovsdb_monitor [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Error received from ovsdb monitor: 2014-07-31T14:42:58Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated)
  2014-07-31 14:42:58.760 28084 DEBUG neutron.agent.linux.async_process [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Halting async process [['ovsdb-client', 'monitor', 'Interface', 'name,ofport', '--format=json']] in response to an error. _handle_process_error /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:173
  2014-07-31 14:42:58.763 28084 DEBUG neutron.agent.linux.utils [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Running command: ['ps', '--ppid', '28160', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:48
  2014-07-31 14:42:58.880 28084 DEBUG neutron.agent.linux.utils [req-0698c817-970d-4c56-bd72-90fb37f1a134 None]
  Command: ['ps', '--ppid', '28160', '-o', 'pid=']
  Exit code: 1
  Stdout: ''
  Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:74
  2014-07-31 14:43:00.040 28084 DEBUG neutron.agent.linux.async_process [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] Halting async process [['ovsdb-client', 'monitor', 'Interface', 'name,ofport', '--format=json']]. stop /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:90
  2014-07-31 14:43:00.052 28084 CRITICAL neutron [req-0698c817-970d-4c56-bd72-90fb37f1a134 None] AssertionError: Trying to re-send() an already-triggered event.
  2014-07-31 14:43:00.052 28084 TRACE neutron Traceback (most recent call last):
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
  2014-07-31 14:43:00.052 28084 TRACE neutron     sys.exit(main())
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1435, in main
  2014-07-31 14:43:00.052 28084 TRACE neutron     agent.daemon_loop()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1363, in daemon_loop
  2014-07-31 14:43:00.052 28084 TRACE neutron     self.rpc_loop(polling_manager=pm)
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2014-07-31 14:43:00.052 28084 TRACE neutron     self.gen.next()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 41, in get_polling_manager
  2014-07-31 14:43:00.052 28084 TRACE neutron     pm.stop()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 108, in stop
  2014-07-31 14:43:00.052 28084 TRACE neutron     self._monitor.stop()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 91, in stop
  2014-07-31 14:43:00.052 28084 TRACE neutron     self._kill()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovsdb_monitor.py", line 108, in _kill
  2014-07-31 14:43:00.052 28084 TRACE neutron     super(SimpleInterfaceMonitor, self)._kill(*args, **kwargs)
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 118, in _kill
  2014-07-31 14:43:00.052 28084 TRACE neutron     self._kill_event.send()
  2014-07-31 14:43:00.052 28084 TRACE neutron   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 150, in send
  2014-07-31 14:43:00.052 28084 TRACE neutron     assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
  2014-07-31 14:43:00.052 28084 TRACE neutron AssertionError: Trying to re-send() an already-triggered event.
  2014-07-31 14:43:00.052 28084 TRACE neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1350903/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next