yahoo-eng-team team mailing list archive

[Assaf Muller <amuller@xxxxxxxxxx>]

Public bug reported:

The default gateway can vanish from the HA router namespace after
certain operations.

My setup:
Fedora 20
keepalived-1.2.13-1.fc20.x86_64
Network manager turned off.

I can reproduce this reliably on my system, but cannot reproduce this on
a RHEL 7 system. Even on that system, the issue manifests on its own, I
just can't reproduce it at will.

How I reproduce on my system:
Create an HA router
Set it as a gateway
Go to the master instance
Observe that the namespace has a default gateway
Add an internal interface (Make sure that the IP is 'lower' than the IP of the external interface, this is explained below)
Default gateway will no longer exist

Cause:
keepalived.conf has two sections for VIPs: virtual_ipaddress, and virtual_ipaddress_excluded. The difference is that any VIPs that go in the first section will be propagated on the wire, and any VIPs in the excluded section do not. Traditional configuration of keepalived places one VIP in the normal section, henceforth known as the 'primary VIP', and all other VIPs in the excluded section. Currently the keepalived manager does this by sorting the VIPs (Internal IPs, external SNAT IP, and all floating IPs), placing the lowest one (By string comparison) as the primary, and the rest of the VIPs in the excluded section: 
https://github.com/openstack/neutron/blob/master/neutron/agent/linux/keepalived.py#L155

That code is ran, and keepalived.conf is built when ever a router is
updated. This means that the primary VIP can change on router updates.
As it turns out, after a conversation with a keepalived developer,
keepalived assumes that the order does not change (This is possibly a
keepalived bug, depending on your view on life, the ordering of the
stars when keepalived is executed and the wind speed in the Falkland
Islands in the past leap year). On my system, with the currently
installed keepalived version, whenever the primary VIP changes, the
default gateway (Present in the virtual_routes section of
keepalived.conf) is violently removed.

Possible solution:
Make sure that the primary VIP never changes. For example: Fabricate an IP per HA router cluster (Derived from the VRID?), add it as a VIP on the HA device, configure it as the primary VIP. I played around with a hacky variation of this solution and I could no longer reproduce the issue.

** Affects: neutron
     Importance: Undecided
     Assignee: Assaf Muller (amuller)
         Status: New


** Tags: juno-backport-potential l3-ha

** Changed in: neutron
     Assignee: (unassigned) => Assaf Muller (amuller)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1404945

Title:
  Default gateway can vanish from HA routers, destroying external
  connectivity for all VMs on that network

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  The default gateway can vanish from the HA router namespace after
  certain operations.

  My setup:
  Fedora 20
  keepalived-1.2.13-1.fc20.x86_64
  Network manager turned off.

  I can reproduce this reliably on my system, but cannot reproduce this
  on a RHEL 7 system. Even on that system, the issue manifests on its
  own, I just can't reproduce it at will.

  How I reproduce on my system:
  Create an HA router
  Set it as a gateway
  Go to the master instance
  Observe that the namespace has a default gateway
  Add an internal interface (Make sure that the IP is 'lower' than the IP of the external interface, this is explained below)
  Default gateway will no longer exist

  Cause:
  keepalived.conf has two sections for VIPs: virtual_ipaddress, and virtual_ipaddress_excluded. The difference is that any VIPs that go in the first section will be propagated on the wire, and any VIPs in the excluded section do not. Traditional configuration of keepalived places one VIP in the normal section, henceforth known as the 'primary VIP', and all other VIPs in the excluded section. Currently the keepalived manager does this by sorting the VIPs (Internal IPs, external SNAT IP, and all floating IPs), placing the lowest one (By string comparison) as the primary, and the rest of the VIPs in the excluded section: 
  https://github.com/openstack/neutron/blob/master/neutron/agent/linux/keepalived.py#L155

  That code is ran, and keepalived.conf is built when ever a router is
  updated. This means that the primary VIP can change on router updates.
  As it turns out, after a conversation with a keepalived developer,
  keepalived assumes that the order does not change (This is possibly a
  keepalived bug, depending on your view on life, the ordering of the
  stars when keepalived is executed and the wind speed in the Falkland
  Islands in the past leap year). On my system, with the currently
  installed keepalived version, whenever the primary VIP changes, the
  default gateway (Present in the virtual_routes section of
  keepalived.conf) is violently removed.

  Possible solution:
  Make sure that the primary VIP never changes. For example: Fabricate an IP per HA router cluster (Derived from the VRID?), add it as a VIP on the HA device, configure it as the primary VIP. I played around with a hacky variation of this solution and I could no longer reproduce the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1404945/+subscriptions