yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1904869] [NEW] neutron-dynamic-routing reported as not alive after adding BGP Peer to BGP Speaker

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mihai Alexandrescu <1904869@xxxxxxxxxxxxxxxxxx>
Date: Thu, 19 Nov 2020 11:19:46 -0000
Reply-to: Bug 1904869 <1904869@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I'm trying to create a PoC setup for neutron-dynamic-routing.
I'm working on an internal company install of OpenStack Queens with components deployed as Docker containers.

I'm an admin for this particular test platform. In terms of compute
nodes I have 1 compute node for VMs and 1 network node (apart from the
control plane nodes).


Reproduction :
I'm installing neutron-dynamic-routing 12.0.1 on top of a neutron base image (which already functions for other Neutron components ;  it also includes ryu 4.15). I'm allowing pip (or easy_install) to install dependencies.
I'm following the guide here :  https://docs.openstack.org/neutron/queens/admin/config-bgp-dynamic-routing.html

Plugin config snippet :
(neutron-bgp-dragent)[root@i1-net0001 /]$ cat /etc/neutron/bgp_dragent.ini
[BGP]
bgp_router_id = 192.168.1.109
bgp_speaker_driver = neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver.RyuBgpDriver

In /etc/neutron/neutron.conf I already set :
service_plugins = router,metering,qos,trunk,segments,neutron.services.firewall.fwaas_plugin.FirewallPlugin,neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin


After adding the BGP Peer to the BGP Speaker, the output of "openstack network agent list --agent-type bgp" reports :  "State UP" but "Alive XXX" .
The agent doesn't report errors.

# openstack network agent list --agent-type bgp
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
| ID                                   | Agent Type                | Host                  | Availability Zone | Alive | State | Binary              |
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
| 2d10cebf-547c-4027-b31c-b62af4af734d | BGP dynamic routing agent | i1-net0001.mgmt.oiaas | None              | :-)   | UP    | neutron-bgp-dragent |
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+

# openstack bgp speaker add peer bgpspeaker bgppeer

# openstack network agent list --agent-type bgp
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
| ID                                   | Agent Type                | Host                  | Availability Zone | Alive | State | Binary              |
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
| 2d10cebf-547c-4027-b31c-b62af4af734d | BGP dynamic routing agent | i1-net0001.mgmt.oiaas | None              | XXX   | UP    | neutron-bgp-dragent |
+--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+


Log messages from dr agent after the "add" command :
"received message with unique_id: 1956299a1cd94c809a1b270abd92670f"
"Lock \"bgp-dr-agent\" acquired by \"neutron_dynamic_routing.services.bgp.agent.bgp_dragent.bgp_peer_association_end\" :: waited 0.000s"
"Received BGP peer associate notification for speaker_id=8e6348c8-d52d-42aa-9075-03940b863c26 peer_id=336dce4d-e0c9-43aa-99ec-f721c1dae10d from the neutron server."
"CALL msg_id: f75f5c50e30a4ffe860a32cab60bc502 exchange 'neutron' topic 'q-bgp-plugin'"
"received reply msg_id: f75f5c50e30a4ffe860a32cab60bc502"
"Calling driver interface for adding BGP peer 192.168.1.10 remote_as=4321 to BGP Speaker running for local_as=1234 auth_type=none"
"API method neighbor.create called with args: {'connect_mode': 'active', 'cap_mbgp_evpn': False, 'remote_as': 4321, 'cap_mbgp_vpnv6': False, 'cap_mbgp_l2vpnfs': False, 'cap_four_octet_as_number': True, 'cap_mbgp_ipv6': False, 'is_next_hop_self': False, 'cap_mbgp_ipv4': True, 'cap_mbgp_ipv4fs': False, 'is_route_reflector_client': False, 'cap_mbgp_ipv6fs': False, 'is_route_server_client': False, 'cap_enhanced_refresh': False, 'peer_next_hop': None, 'password': None, 'ip_address': u'192.168.1.10', 'cap_mbgp_vpnv4fs': False, 'cap_mbgp_vpnv4': False, 'cap_mbgp_vpnv6fs': False}"
"Create CustomEvent called"
"Create CustomEvent called"
"Added BGP Peer 192.168.1.10 for remote_as=4321 to BGP Speaker running for local_as=1234."
"Lock \"bgp-dr-agent\" released by \"neutron_dynamic_routing.services.bgp.agent.bgp_dragent.bgp_peer_association_end\" :: held 0.059s"


After this, neutron-api logs a few times:
"Agent healthcheck: found 1 dead agents out of 6:\n                Type       Last heartbeat host\nBGP dynamic routing agent  2020-11-19 10:58:17 i1-net0001.mgmt.oiaas"


Problem is that after that "add", the container doesn't stop working but the agent doesn't log anything else and requires a restart to be considered Alive again.
The dr agent is also not generating any BGP messages (not even TCP session attempts to the neighbor). I have connectivity with the neighbor IP.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1904869

Title:
  neutron-dynamic-routing reported as not alive after adding BGP Peer to
  BGP Speaker

Status in neutron:
  New

Bug description:
  I'm trying to create a PoC setup for neutron-dynamic-routing.
  I'm working on an internal company install of OpenStack Queens with components deployed as Docker containers.

  I'm an admin for this particular test platform. In terms of compute
  nodes I have 1 compute node for VMs and 1 network node (apart from the
  control plane nodes).

  
  Reproduction :
  I'm installing neutron-dynamic-routing 12.0.1 on top of a neutron base image (which already functions for other Neutron components ;  it also includes ryu 4.15). I'm allowing pip (or easy_install) to install dependencies.
  I'm following the guide here :  https://docs.openstack.org/neutron/queens/admin/config-bgp-dynamic-routing.html

  Plugin config snippet :
  (neutron-bgp-dragent)[root@i1-net0001 /]$ cat /etc/neutron/bgp_dragent.ini
  [BGP]
  bgp_router_id = 192.168.1.109
  bgp_speaker_driver = neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver.RyuBgpDriver

  In /etc/neutron/neutron.conf I already set :
  service_plugins = router,metering,qos,trunk,segments,neutron.services.firewall.fwaas_plugin.FirewallPlugin,neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin


  After adding the BGP Peer to the BGP Speaker, the output of "openstack network agent list --agent-type bgp" reports :  "State UP" but "Alive XXX" .
  The agent doesn't report errors.

  # openstack network agent list --agent-type bgp
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
  | ID                                   | Agent Type                | Host                  | Availability Zone | Alive | State | Binary              |
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
  | 2d10cebf-547c-4027-b31c-b62af4af734d | BGP dynamic routing agent | i1-net0001.mgmt.oiaas | None              | :-)   | UP    | neutron-bgp-dragent |
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+

  # openstack bgp speaker add peer bgpspeaker bgppeer

  # openstack network agent list --agent-type bgp
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
  | ID                                   | Agent Type                | Host                  | Availability Zone | Alive | State | Binary              |
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+
  | 2d10cebf-547c-4027-b31c-b62af4af734d | BGP dynamic routing agent | i1-net0001.mgmt.oiaas | None              | XXX   | UP    | neutron-bgp-dragent |
  +--------------------------------------+---------------------------+-----------------------+-------------------+-------+-------+---------------------+


  
  Log messages from dr agent after the "add" command :
  "received message with unique_id: 1956299a1cd94c809a1b270abd92670f"
  "Lock \"bgp-dr-agent\" acquired by \"neutron_dynamic_routing.services.bgp.agent.bgp_dragent.bgp_peer_association_end\" :: waited 0.000s"
  "Received BGP peer associate notification for speaker_id=8e6348c8-d52d-42aa-9075-03940b863c26 peer_id=336dce4d-e0c9-43aa-99ec-f721c1dae10d from the neutron server."
  "CALL msg_id: f75f5c50e30a4ffe860a32cab60bc502 exchange 'neutron' topic 'q-bgp-plugin'"
  "received reply msg_id: f75f5c50e30a4ffe860a32cab60bc502"
  "Calling driver interface for adding BGP peer 192.168.1.10 remote_as=4321 to BGP Speaker running for local_as=1234 auth_type=none"
  "API method neighbor.create called with args: {'connect_mode': 'active', 'cap_mbgp_evpn': False, 'remote_as': 4321, 'cap_mbgp_vpnv6': False, 'cap_mbgp_l2vpnfs': False, 'cap_four_octet_as_number': True, 'cap_mbgp_ipv6': False, 'is_next_hop_self': False, 'cap_mbgp_ipv4': True, 'cap_mbgp_ipv4fs': False, 'is_route_reflector_client': False, 'cap_mbgp_ipv6fs': False, 'is_route_server_client': False, 'cap_enhanced_refresh': False, 'peer_next_hop': None, 'password': None, 'ip_address': u'192.168.1.10', 'cap_mbgp_vpnv4fs': False, 'cap_mbgp_vpnv4': False, 'cap_mbgp_vpnv6fs': False}"
  "Create CustomEvent called"
  "Create CustomEvent called"
  "Added BGP Peer 192.168.1.10 for remote_as=4321 to BGP Speaker running for local_as=1234."
  "Lock \"bgp-dr-agent\" released by \"neutron_dynamic_routing.services.bgp.agent.bgp_dragent.bgp_peer_association_end\" :: held 0.059s"

  
  After this, neutron-api logs a few times:
  "Agent healthcheck: found 1 dead agents out of 6:\n                Type       Last heartbeat host\nBGP dynamic routing agent  2020-11-19 10:58:17 i1-net0001.mgmt.oiaas"


  Problem is that after that "add", the container doesn't stop working but the agent doesn't log anything else and requires a restart to be considered Alive again.
  The dr agent is also not generating any BGP messages (not even TCP session attempts to the neighbor). I have connectivity with the neighbor IP.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1904869/+subscriptions
Follow ups

[Bug 1904869] Re: neutron-dynamic-routing reported as not alive after adding BGP Peer to BGP Speaker
From: Launchpad Bug Tracker, 2021-01-24