yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1675187] [NEW] Floating IPs not removed on rfp interface in qrouter

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: xagent <xagent@xxxxxxxxx>
Date: Wed, 22 Mar 2017 21:37:01 -0000
Reply-to: Bug 1675187 <1675187@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Recently upgraded from liberty to newton. We had a lot of Active
Floating IPs configured at the time.

DVR setup, with 2 VMs on same network, and both with Floating IPs, FIPA
and FIPB.

 - ssh into both FIPA and FIPB works from external source like laptop
 - SSH from one VM into another works via internal fixed IP ONLY(for example, ssh into floating of VMA, then ssh into fixed IP of VM B, or vice versa)
 - ping from one VM to floating IP of other *appears* to work. But even after deleting VM, pings continued. I suspect the rfp interface is responding to ICMP since it has FIP address configured

Noticed that qrouter contained several /32 FIP addresses configured on
rfp interface, but new Floating IPs we created were not being added as
secondary IP addresses.

Fix for bug https://bugs.launchpad.net/neutron/+bug/1462154 is
responsible, I suspect. Note: this was backported into mitaka, so same
issue should exist for liberty->mitaka upgrade.

It removed the logic to both add and remove Floating IP on rfp - now the
add/remove_floating_ip logic only handles the routes in the fip
namespace, source ip rules in qrouter, etc.... The new agent is
basically unaware of any floating IPs configured on the rfp interface.

So it seems any pre-existing FIPs added as secondary IP address in
qrouter remain as zombies. Attaching/reattahing, or deleting and re-
creating does not remove these IPs as the logic to remove was also gone!
Please see:
https://review.openstack.org/#/c/289172/13/neutron/agent/l3/dvr_local_router.py

As you can see below, lot of FIPs on 10.4.0.0/16 external network. Some
correspond to VMs/floating IPs that were deleted. Others are still
active but experiencing issue described above.

Manually removing the IP immediately fixed our issue (ssh to floating IP
from VM to VM worked immediately)

root@barney:~# ip netns exec qrouter-37176403-cfb0-478d-b51c-971d89597cf5 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-37176403-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 76:2f:be:73:9b:fc brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.142/31 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.253.15/32 brd 10.4.253.15 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.253.118/32 brd 10.4.253.118 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.252.103/32 brd 10.4.252.103 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.252.105/32 brd 10.4.252.105 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.254.4/32 brd 10.4.254.4 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.254.52/32 brd 10.4.254.52 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.253.41/32 brd 10.4.253.41 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.254.228/32 brd 10.4.254.228 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.254.229/32 brd 10.4.254.229 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.253.212/32 brd 10.4.253.212 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.252.118/32 brd 10.4.252.118 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet 10.4.252.48/32 brd 10.4.252.48 scope global rfp-37176403-c
       valid_lft forever preferred_lft forever
    inet6 fe80::742f:beff:fe73:9bfc/64 scope link 
       valid_lft forever preferred_lft forever
11: qr-43981e59-30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:42:a2:ec brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/16 brd 172.16.255.255 scope global qr-43981e59-30
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe42:a2ec/64 scope link 
       valid_lft forever preferred_lft forever
40083: qr-6b09eb9d-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:21:f8:fd brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.100/24 brd 192.168.42.255 scope global qr-6b09eb9d-40
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe21:f8fd/64 scope link 
       valid_lft forever preferred_lft forever
46229: qr-39850654-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:f5:06:20 brd ff:ff:ff:ff:ff:ff
    inet 10.127.0.1/16 brd 10.127.255.255 scope global qr-39850654-40
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fef5:620/64 scope link 
       valid_lft forever preferred_lft forever

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1675187

Title:
  Floating IPs not removed on rfp interface in qrouter

Status in neutron:
  New

Bug description:
  Recently upgraded from liberty to newton. We had a lot of Active
  Floating IPs configured at the time.

  DVR setup, with 2 VMs on same network, and both with Floating IPs,
  FIPA and FIPB.

   - ssh into both FIPA and FIPB works from external source like laptop
   - SSH from one VM into another works via internal fixed IP ONLY(for example, ssh into floating of VMA, then ssh into fixed IP of VM B, or vice versa)
   - ping from one VM to floating IP of other *appears* to work. But even after deleting VM, pings continued. I suspect the rfp interface is responding to ICMP since it has FIP address configured

  Noticed that qrouter contained several /32 FIP addresses configured on
  rfp interface, but new Floating IPs we created were not being added as
  secondary IP addresses.

  Fix for bug https://bugs.launchpad.net/neutron/+bug/1462154 is
  responsible, I suspect. Note: this was backported into mitaka, so same
  issue should exist for liberty->mitaka upgrade.

  It removed the logic to both add and remove Floating IP on rfp - now
  the add/remove_floating_ip logic only handles the routes in the fip
  namespace, source ip rules in qrouter, etc.... The new agent is
  basically unaware of any floating IPs configured on the rfp interface.

  So it seems any pre-existing FIPs added as secondary IP address in
  qrouter remain as zombies. Attaching/reattahing, or deleting and re-
  creating does not remove these IPs as the logic to remove was also
  gone! Please see:
  https://review.openstack.org/#/c/289172/13/neutron/agent/l3/dvr_local_router.py

  As you can see below, lot of FIPs on 10.4.0.0/16 external network.
  Some correspond to VMs/floating IPs that were deleted. Others are
  still active but experiencing issue described above.

  Manually removing the IP immediately fixed our issue (ssh to floating
  IP from VM to VM worked immediately)

  root@barney:~# ip netns exec qrouter-37176403-cfb0-478d-b51c-971d89597cf5 ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
  2: rfp-37176403-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether 76:2f:be:73:9b:fc brd ff:ff:ff:ff:ff:ff
      inet 169.254.31.142/31 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.15/32 brd 10.4.253.15 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.118/32 brd 10.4.253.118 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.103/32 brd 10.4.252.103 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.105/32 brd 10.4.252.105 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.4/32 brd 10.4.254.4 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.52/32 brd 10.4.254.52 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.41/32 brd 10.4.253.41 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.228/32 brd 10.4.254.228 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.229/32 brd 10.4.254.229 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.212/32 brd 10.4.253.212 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.118/32 brd 10.4.252.118 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.48/32 brd 10.4.252.48 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet6 fe80::742f:beff:fe73:9bfc/64 scope link 
         valid_lft forever preferred_lft forever
  11: qr-43981e59-30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:42:a2:ec brd ff:ff:ff:ff:ff:ff
      inet 172.16.0.1/16 brd 172.16.255.255 scope global qr-43981e59-30
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe42:a2ec/64 scope link 
         valid_lft forever preferred_lft forever
  40083: qr-6b09eb9d-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:21:f8:fd brd ff:ff:ff:ff:ff:ff
      inet 192.168.42.100/24 brd 192.168.42.255 scope global qr-6b09eb9d-40
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe21:f8fd/64 scope link 
         valid_lft forever preferred_lft forever
  46229: qr-39850654-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:f5:06:20 brd ff:ff:ff:ff:ff:ff
      inet 10.127.0.1/16 brd 10.127.255.255 scope global qr-39850654-40
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fef5:620/64 scope link 
         valid_lft forever preferred_lft forever

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1675187/+subscriptions
Follow ups

[Bug 1675187] Re: Floating IPs not removed on rfp interface in qrouter
From: OpenStack Infra, 2017-04-02