yahoo-eng-team team mailing list archive

[Ryan Tidwell <1820297@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I'm observing backtraces in neutron-dynamic-routing scenario job logs.
The neutron dynamic routing agent is logging the following, seemingly
after a BGP peer goes idle

Feb 26 13:07:55.571869 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: INFO neutron_dynamic_routing.services.bgp.agent.driver.os_ken.driver [-] BGP Peer 192.168.10.129 for remote_as=64522 went DOWN.
Feb 26 13:07:55.572107 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: DEBUG bgpspeaker.peer [-] Peer 192.168.10.129 BGP FSM went from Established to Idle {{(pid=21925) bgp_state /usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/peer.py:237}}
Feb 26 13:07:55.572335 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: ERROR bgpspeaker.base [-] Traceback (most recent call last):
Feb 26 13:07:55.572545 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/base.py", line 256, in start
Feb 26 13:07:55.572762 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._run(*args, **kwargs)
Feb 26 13:07:55.572969 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 275, in _run
Feb 26 13:07:55.573167 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._recv_loop()
Feb 26 13:07:55.573375 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 571, in _recv_loop
Feb 26 13:07:55.573581 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self.connection_lost(conn_lost_reason)
Feb 26 13:07:55.573786 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 596, in connection_lost
Feb 26 13:07:55.573992 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._peer.connection_lost(reason)
Feb 26 13:07:55.574191 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/peer.py", line 2323, in connection_lost
Feb 26 13:07:55.574391 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._protocol.stop()
Feb 26 13:07:55.574590 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 405, in stop
Feb 26 13:07:55.574797 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     Activity.stop(self)
Feb 26 13:07:55.574996 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/base.py", line 314, in stop
Feb 26 13:07:55.575208 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     raise ActivityException(desc='Cannot call stop when activity is '
Feb 26 13:07:55.575406 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: ActivityException: 100.1 - Cannot call stop when activity is not started or has been stopped already.
Feb 26 13:07:55.575639 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: : ActivityException: 100.1 - Cannot call stop when activity is not started or has been stopped already.

I'm not sure of the impact yet. At a minimum, it would be nice to see a
nice log message instead of a backtrace (assuming this is doesn't
indicate a real issue).

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1820297

Title:
  os-ken error when BGP peer goes down

Status in neutron:
  New

Bug description:
  I'm observing backtraces in neutron-dynamic-routing scenario job logs.
  The neutron dynamic routing agent is logging the following, seemingly
  after a BGP peer goes idle

  Feb 26 13:07:55.571869 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: INFO neutron_dynamic_routing.services.bgp.agent.driver.os_ken.driver [-] BGP Peer 192.168.10.129 for remote_as=64522 went DOWN.
  Feb 26 13:07:55.572107 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: DEBUG bgpspeaker.peer [-] Peer 192.168.10.129 BGP FSM went from Established to Idle {{(pid=21925) bgp_state /usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/peer.py:237}}
  Feb 26 13:07:55.572335 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: ERROR bgpspeaker.base [-] Traceback (most recent call last):
  Feb 26 13:07:55.572545 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/base.py", line 256, in start
  Feb 26 13:07:55.572762 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._run(*args, **kwargs)
  Feb 26 13:07:55.572969 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 275, in _run
  Feb 26 13:07:55.573167 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._recv_loop()
  Feb 26 13:07:55.573375 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 571, in _recv_loop
  Feb 26 13:07:55.573581 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self.connection_lost(conn_lost_reason)
  Feb 26 13:07:55.573786 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 596, in connection_lost
  Feb 26 13:07:55.573992 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._peer.connection_lost(reason)
  Feb 26 13:07:55.574191 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/peer.py", line 2323, in connection_lost
  Feb 26 13:07:55.574391 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     self._protocol.stop()
  Feb 26 13:07:55.574590 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 405, in stop
  Feb 26 13:07:55.574797 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     Activity.stop(self)
  Feb 26 13:07:55.574996 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:   File "/usr/local/lib/python2.7/dist-packages/os_ken/services/protocols/bgp/base.py", line 314, in stop
  Feb 26 13:07:55.575208 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]:     raise ActivityException(desc='Cannot call stop when activity is '
  Feb 26 13:07:55.575406 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: ActivityException: 100.1 - Cannot call stop when activity is not started or has been stopped already.
  Feb 26 13:07:55.575639 ubuntu-xenial-inap-mtl01-0003035225 neutron-bgp-dragent[21925]: : ActivityException: 100.1 - Cannot call stop when activity is not started or has been stopped already.

  I'm not sure of the impact yet. At a minimum, it would be nice to see
  a nice log message instead of a backtrace (assuming this is doesn't
  indicate a real issue).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1820297/+subscriptions