yahoo-eng-team team mailing list archive

[Alexander Gräb <1890400@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

With Rocky commit f2d234e introduces a change making Neutron more
compatible with Keepalived 2.x. A new option `no_track` is passed to
Keepalived by the L3 agent which Keepalived 1.x doesn't recognize, thus
the configuration is partially not applied by Keepalived 1.x. resulting
in a missing default gateway. Thus instances using HA routers to
communicate with the outside world are unable to reach the internet and
cannot be reached via their floating IP addresses.

There are some workarounds to trigger the creating of the default
gateway like disable and enable the router again or disable/re-start the
L3 agent which hosts the master namespace of the HA router.

Steps to reproduce:
1. Create an HA router
2. Add the gateway network (now the default gateway should be set within the routers master network namespace but there is none)
3. Connect a tenant subnet to the router
4. Create an instance connected to the tenant network created in step 3
5. Try to reach the internet from within the instances created in step 4

I was able to get some log output out of Keepalived:
Tue Aug  4 11:59:58 2020: Starting Keepalived v1.3.9 (10/21,2017)
Tue Aug  4 11:59:58 2020: Opening file '/var/lib/neutron/ha_confs/019f8036-8730-4584-9e07-c4a6504447ab/keepalived.conf'.
Tue Aug  4 11:59:58 2020: Starting VRRP child process, pid=2864
Tue Aug  4 11:59:58 2020: Registering Kernel netlink reflector
Tue Aug  4 11:59:58 2020: Registering Kernel netlink command channel
Tue Aug  4 11:59:58 2020: Registering gratuitous ARP shared channel
Tue Aug  4 11:59:58 2020: Opening file '/var/lib/neutron/ha_confs/019f8036-8730-4584-9e07-c4a6504447ab/keepalived.conf'.
Tue Aug  4 11:59:58 2020: Unknown configuration entry 'no_track' for ip address - ignoring
Tue Aug  4 11:59:58 2020: Unknown configuration entry 'no_track' for ip address - ignoring
Tue Aug  4 11:59:58 2020: Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe2c:6622/64) - ignoring scope
Tue Aug  4 11:59:58 2020: VRRP parsed invalid IP no_track. skipping IP...
Tue Aug  4 11:59:58 2020: unknown route keyword no_track
Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) removing protocol VIPs.
Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) removing protocol E-VIPs.
Tue Aug  4 11:59:58 2020: Using LinkWatch kernel netlink reflector...
Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) Entering BACKUP STATE
Tue Aug  4 11:59:58 2020: VRRP sockpool: [ifindex(1033), proto(112), unicast(0), fd(9,10)]

You see it complaining about the 'no_track' option.

We use Kolla containers with Ubuntu base. Even though Keepalived 2 was
release quite a while ago, it still only provides Keepalived 1.x via the
package repositories. Even Kolla in the latest version still uses Ubuntu
18.04 as base with Keepalived 1.x. Theoretically all users using Kolla
containers with Ubuntu base (other base images not tested) are affected.
There seems to be no apt sources for Keepalived 2.x for Ubuntu 18.04.
You need to compile it from source in order to get a newer version.

Maybe it should depend on the Keepalived version whether to pass the
'no_track' option or make it configurable.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-ha

** Tags added: l3-ha

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1890400

Title:
  Default gateway in HA router namespace not set if using Keepalived 1.x

Status in neutron:
  New

Bug description:
  With Rocky commit f2d234e introduces a change making Neutron more
  compatible with Keepalived 2.x. A new option `no_track` is passed to
  Keepalived by the L3 agent which Keepalived 1.x doesn't recognize,
  thus the configuration is partially not applied by Keepalived 1.x.
  resulting in a missing default gateway. Thus instances using HA
  routers to communicate with the outside world are unable to reach the
  internet and cannot be reached via their floating IP addresses.

  There are some workarounds to trigger the creating of the default
  gateway like disable and enable the router again or disable/re-start
  the L3 agent which hosts the master namespace of the HA router.

  Steps to reproduce:
  1. Create an HA router
  2. Add the gateway network (now the default gateway should be set within the routers master network namespace but there is none)
  3. Connect a tenant subnet to the router
  4. Create an instance connected to the tenant network created in step 3
  5. Try to reach the internet from within the instances created in step 4

  I was able to get some log output out of Keepalived:
  Tue Aug  4 11:59:58 2020: Starting Keepalived v1.3.9 (10/21,2017)
  Tue Aug  4 11:59:58 2020: Opening file '/var/lib/neutron/ha_confs/019f8036-8730-4584-9e07-c4a6504447ab/keepalived.conf'.
  Tue Aug  4 11:59:58 2020: Starting VRRP child process, pid=2864
  Tue Aug  4 11:59:58 2020: Registering Kernel netlink reflector
  Tue Aug  4 11:59:58 2020: Registering Kernel netlink command channel
  Tue Aug  4 11:59:58 2020: Registering gratuitous ARP shared channel
  Tue Aug  4 11:59:58 2020: Opening file '/var/lib/neutron/ha_confs/019f8036-8730-4584-9e07-c4a6504447ab/keepalived.conf'.
  Tue Aug  4 11:59:58 2020: Unknown configuration entry 'no_track' for ip address - ignoring
  Tue Aug  4 11:59:58 2020: Unknown configuration entry 'no_track' for ip address - ignoring
  Tue Aug  4 11:59:58 2020: Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe2c:6622/64) - ignoring scope
  Tue Aug  4 11:59:58 2020: VRRP parsed invalid IP no_track. skipping IP...
  Tue Aug  4 11:59:58 2020: unknown route keyword no_track
  Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) removing protocol VIPs.
  Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) removing protocol E-VIPs.
  Tue Aug  4 11:59:58 2020: Using LinkWatch kernel netlink reflector...
  Tue Aug  4 11:59:58 2020: VRRP_Instance(VR_23) Entering BACKUP STATE
  Tue Aug  4 11:59:58 2020: VRRP sockpool: [ifindex(1033), proto(112), unicast(0), fd(9,10)]

  You see it complaining about the 'no_track' option.

  We use Kolla containers with Ubuntu base. Even though Keepalived 2 was
  release quite a while ago, it still only provides Keepalived 1.x via
  the package repositories. Even Kolla in the latest version still uses
  Ubuntu 18.04 as base with Keepalived 1.x. Theoretically all users
  using Kolla containers with Ubuntu base (other base images not tested)
  are affected. There seems to be no apt sources for Keepalived 2.x for
  Ubuntu 18.04. You need to compile it from source in order to get a
  newer version.

  Maybe it should depend on the Keepalived version whether to pass the
  'no_track' option or make it configurable.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1890400/+subscriptions