← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #92510

[Bug 2024381] [NEW] keepalived fails to start after updating DVR-HA internal network MTU

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

We got an issue when keepalived stops to be running after update MTU on the internal network of the DVR-HA router.
It turned out that the keepalived config has an interface from qrouter-ns although the keepalived process itself is running in snat-ns.

Here is a simple demo on the latest master branch:
$ openstack network create net1
$ openstack subnet create sub1 --network net1 --subnet-range 192.168.100.0/24
$ openstack router create r1 --distributed --ha
$ openstack router add subnet r1 sub1

Keepalived process is running and the config looks like:

$ ps axf | grep -w pid.keepalived
...
 130250 ?        S      0:00  \_ keepalived -P -f /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf -p /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd.pid.keepalived -r /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd.pid.keepalived-vrrp -D
$ cat /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
global_defs {
    notification_email_from neutron@openstack.local
    router_id neutron
}
vrrp_instance VR_60 {
    state BACKUP
    interface ha-77ee55dc-5c
    virtual_router_id 60
    priority 50
    garp_master_delay 60
    nopreempt
    advert_int 2
    track_interface {
        ha-77ee55dc-5c
    }
    virtual_ipaddress {
        169.254.0.60/24 dev ha-77ee55dc-5c
    }
$


Now update MTU of the internal network:

$ openstack network set net1 --mtu 1400
$ ps axf | grep -w pid.keepalived
 131097 pts/0    S+     0:00  |           \_ grep --color=auto -w pid.keepalived
$ 

$ ip netns exec snat-f7df848f-f168-4305-8ba2-a31902bdbbfd keepalived -t -f /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
(/opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf: Line 20) WARNING - interface qr-035f8095-76 for ip address 192.168.100.1/24 doesn't exist
(/opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf: Line 21) WARNING - interface qr-035f8095-76 for ip address fe80::f816:3eff:fe88:e922/64 doesn't exist
Non-existent interface specified in configuration
$
$ cat /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
global_defs {
    notification_email_from neutron@openstack.local
    router_id neutron
}
vrrp_instance VR_60 {
    state BACKUP
    interface ha-77ee55dc-5c
    virtual_router_id 60
    priority 50
    garp_master_delay 60
    nopreempt
    advert_int 2
    track_interface {
        ha-77ee55dc-5c
    }
    virtual_ipaddress {
        169.254.0.60/24 dev ha-77ee55dc-5c
    }
    virtual_ipaddress_excluded {
        192.168.100.1/24 dev qr-035f8095-76
        fe80::f816:3eff:fe88:e922/64 dev qr-035f8095-76 scope link
    }
}$

$ ip netns exec snat-f7df848f-f168-4305-8ba2-a31902bdbbfd ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: ha-77ee55dc-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:46:30:c4 brd ff:ff:ff:ff:ff:ff
$

** Affects: neutron
     Importance: Undecided
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2024381

Title:
  keepalived fails to start after updating DVR-HA internal network MTU

Status in neutron:
  In Progress

Bug description:
  We got an issue when keepalived stops to be running after update MTU on the internal network of the DVR-HA router.
  It turned out that the keepalived config has an interface from qrouter-ns although the keepalived process itself is running in snat-ns.

  Here is a simple demo on the latest master branch:
  $ openstack network create net1
  $ openstack subnet create sub1 --network net1 --subnet-range 192.168.100.0/24
  $ openstack router create r1 --distributed --ha
  $ openstack router add subnet r1 sub1

  Keepalived process is running and the config looks like:

  $ ps axf | grep -w pid.keepalived
  ...
   130250 ?        S      0:00  \_ keepalived -P -f /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf -p /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd.pid.keepalived -r /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd.pid.keepalived-vrrp -D
  $ cat /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
  global_defs {
      notification_email_from neutron@openstack.local
      router_id neutron
  }
  vrrp_instance VR_60 {
      state BACKUP
      interface ha-77ee55dc-5c
      virtual_router_id 60
      priority 50
      garp_master_delay 60
      nopreempt
      advert_int 2
      track_interface {
          ha-77ee55dc-5c
      }
      virtual_ipaddress {
          169.254.0.60/24 dev ha-77ee55dc-5c
      }
  $

  
  Now update MTU of the internal network:

  $ openstack network set net1 --mtu 1400
  $ ps axf | grep -w pid.keepalived
   131097 pts/0    S+     0:00  |           \_ grep --color=auto -w pid.keepalived
  $ 

  $ ip netns exec snat-f7df848f-f168-4305-8ba2-a31902bdbbfd keepalived -t -f /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
  (/opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf: Line 20) WARNING - interface qr-035f8095-76 for ip address 192.168.100.1/24 doesn't exist
  (/opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf: Line 21) WARNING - interface qr-035f8095-76 for ip address fe80::f816:3eff:fe88:e922/64 doesn't exist
  Non-existent interface specified in configuration
  $
  $ cat /opt/stack/data/neutron/ha_confs/f7df848f-f168-4305-8ba2-a31902bdbbfd/keepalived.conf
  global_defs {
      notification_email_from neutron@openstack.local
      router_id neutron
  }
  vrrp_instance VR_60 {
      state BACKUP
      interface ha-77ee55dc-5c
      virtual_router_id 60
      priority 50
      garp_master_delay 60
      nopreempt
      advert_int 2
      track_interface {
          ha-77ee55dc-5c
      }
      virtual_ipaddress {
          169.254.0.60/24 dev ha-77ee55dc-5c
      }
      virtual_ipaddress_excluded {
          192.168.100.1/24 dev qr-035f8095-76
          fe80::f816:3eff:fe88:e922/64 dev qr-035f8095-76 scope link
      }
  }$

  $ ip netns exec snat-f7df848f-f168-4305-8ba2-a31902bdbbfd ip link
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  10: ha-77ee55dc-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/ether fa:16:3e:46:30:c4 brd ff:ff:ff:ff:ff:ff
  $

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2024381/+subscriptions
  Thread Previous • Date Previous • • Thread Next