yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1874211] [NEW] [L3HA] Keepalived 2.x.x tracks state of virtual_ipaddresses interfaces and router now

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1874211@xxxxxxxxxxxxxxxxxx>
Date: Wed, 22 Apr 2020 08:21:08 -0000
Reply-to: Bug 1874211 <1874211@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Patch https://review.opendev.org/#/c/707406/ introduced new mechanism which brings all HA router's interfaces to be DOWN if router is in backup mode.
And that works fine with keepalived 1.4.x but in keepalived 2.x.x it changed and keepalived now tracks by default interfaces of virtual_ipaddresses and routes. And will go to FAULT state if such interface is DOWN.
That cause problem with router which will never be transitioned to master state.

We should add "no_track" option to qg- and qr- interfaces in keepalived
config file now.

Errors can be seen e.g. in result of tripleo job:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2b3/721574/1/check
/tripleo-ci-
centos-8-scenario007-standalone/2b3f794/logs/undercloud/var/log/journal.txt

Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting Keepalived v2.0.10 (11/12,2018)
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Running on Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 (built for Linux 4.18.0)
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Command line: '/usr/sbin/keepalived' '-n' '-l' '-D' '-P' '-f'
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf' '-p'
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived' '-r'
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived-vrrp'
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '-D'
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'.
Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting VRRP child process, pid=147864
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink reflector
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink command channel
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'.
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 22) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe0a:3675/64) - ignoring scope
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 23) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fea8:ae56/64) - ignoring scope
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) Ignoring track_interface ha-dcefeeaa-6d since own interface
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address 169.254.195.203 for interface ha-dcefeeaa-6d
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address fe80::f816:3eff:fec7:3b26 for interface ha-dcefeeaa-6d
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) entering FAULT state
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous ARP shared channel
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous NDISC shared channel
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing VIPs.
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing E-VIPs.
Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes

** Affects: neutron
     Importance: Critical
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1874211

Title:
  [L3HA] Keepalived 2.x.x tracks state of virtual_ipaddresses interfaces
  and router now

Status in neutron:
  Confirmed

Bug description:
  Patch https://review.opendev.org/#/c/707406/ introduced new mechanism which brings all HA router's interfaces to be DOWN if router is in backup mode.
  And that works fine with keepalived 1.4.x but in keepalived 2.x.x it changed and keepalived now tracks by default interfaces of virtual_ipaddresses and routes. And will go to FAULT state if such interface is DOWN.
  That cause problem with router which will never be transitioned to master state.

  We should add "no_track" option to qg- and qr- interfaces in
  keepalived config file now.

  Errors can be seen e.g. in result of tripleo job:
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2b3/721574/1/check
  /tripleo-ci-
  centos-8-scenario007-standalone/2b3f794/logs/undercloud/var/log/journal.txt

  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting Keepalived v2.0.10 (11/12,2018)
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Running on Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 (built for Linux 4.18.0)
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Command line: '/usr/sbin/keepalived' '-n' '-l' '-D' '-P' '-f'
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf' '-p'
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived' '-r'
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c.pid.keepalived-vrrp'
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]:               '-D'
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'.
  Apr 21 12:21:45 standalone.localdomain Keepalived[147861]: Starting VRRP child process, pid=147864
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink reflector
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering Kernel netlink command channel
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Opening file '/var/lib/neutron/ha_confs/24a8996a-5d64-446d-afcd-e08c3d72d64c/keepalived.conf'.
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 22) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe0a:3675/64) - ignoring scope
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (Line 23) Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fea8:ae56/64) - ignoring scope
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) Ignoring track_interface ha-dcefeeaa-6d since own interface
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address 169.254.195.203 for interface ha-dcefeeaa-6d
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Assigned address fe80::f816:3eff:fec7:3b26 for interface ha-dcefeeaa-6d
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) entering FAULT state
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous ARP shared channel
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: Registering gratuitous NDISC shared channel
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing VIPs.
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing E-VIPs.
  Apr 21 12:21:45 standalone.localdomain Keepalived_vrrp[147864]: (VR_99) removing Virtual Routes

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1874211/+subscriptions
Follow ups

[Bug 1874211] Re: [L3HA] Keepalived 2.x.x tracks state of virtual_ipaddresses interfaces and router now
From: OpenStack Infra, 2020-04-23