yahoo-eng-team team mailing list archive

[Andrew Bonney <2049909@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Neutron 2023.1 29cc1a634e530972614c09fbb212b5f63fd4c374
Ubuntu 20.04

This issue has been identified in a Neutron system running Linux Bridge
networking, but whilst this may no longer be supported I'm posting it in
case the same issue might be relevant for other drivers.

When running multiple network nodes, the tenant networks in the
namespaces on each node share MAC addresses. We have noted that
particularly when rebooting a network node, traffic from the Internet to
tenant networks can be disrupted when a node which was acting as the
backup for a given tenant network comes back online. We have traced this
to Linux sending out MLDv2 responses to the upstream switches when the
tenant network processes (keepalived etc) start up. As a result, the
upstream switches update their MAC tables to prefer that host despite it
not being the primary. If there is minimal tenant traffic (such as when
running a web server), this network will be inaccessible from the
outside until a request is made from the inside to the outside and the
switches re-update their MAC tables to reflect the correct state.

There is already handling to prevent some IPv6 packets being sent out in
these cases here:
https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L808,
and there is theoretically something explicitly referencing issues with
MLDv2 in the same area:
https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L813.
Unfortunately these don't appear to be sufficient.

There is no sysctl mechanism to prevent these gratuitous MLDv2 responses
as far as I can tell, so we are working around this by using iptables
rules inserted by the L3 agent into tenant networks. There may well be a
better solution, but I will link our workaround to this bug report
shortly.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2049909

Title:
  MLDv2 packets sent from L3 Agent managed networks cause backup routers
  to be preferred

Status in neutron:
  New

Bug description:
  Neutron 2023.1 29cc1a634e530972614c09fbb212b5f63fd4c374
  Ubuntu 20.04

  This issue has been identified in a Neutron system running Linux
  Bridge networking, but whilst this may no longer be supported I'm
  posting it in case the same issue might be relevant for other drivers.

  When running multiple network nodes, the tenant networks in the
  namespaces on each node share MAC addresses. We have noted that
  particularly when rebooting a network node, traffic from the Internet
  to tenant networks can be disrupted when a node which was acting as
  the backup for a given tenant network comes back online. We have
  traced this to Linux sending out MLDv2 responses to the upstream
  switches when the tenant network processes (keepalived etc) start up.
  As a result, the upstream switches update their MAC tables to prefer
  that host despite it not being the primary. If there is minimal tenant
  traffic (such as when running a web server), this network will be
  inaccessible from the outside until a request is made from the inside
  to the outside and the switches re-update their MAC tables to reflect
  the correct state.

  There is already handling to prevent some IPv6 packets being sent out
  in these cases here:
  https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L808,
  and there is theoretically something explicitly referencing issues
  with MLDv2 in the same area:
  https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/router_info.py#L813.
  Unfortunately these don't appear to be sufficient.

  There is no sysctl mechanism to prevent these gratuitous MLDv2
  responses as far as I can tell, so we are working around this by using
  iptables rules inserted by the L3 agent into tenant networks. There
  may well be a better solution, but I will link our workaround to this
  bug report shortly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2049909/+subscriptions