yahoo-eng-team team mailing list archive

[OpenStack Infra <1675187@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/451859
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f84ce9a57ceaf6f931ca51e028bac64c93835d91
Submitter: Jenkins
Branch:    master

commit f84ce9a57ceaf6f931ca51e028bac64c93835d91
Author: Brian Haley <bhaley@xxxxxxxxxx>
Date:   Wed Mar 29 16:44:57 2017 -0400

    Remove stale floating IP addresses from rfp devices
    
    Old versions of the DVR code used to configure floating
    IP addresses directly on the rfp-* devices in the
    qrouter namespace.  This was changed in Mitaka to use
    routes, but these stale floating IP addresses were
    never cleaned-up when the l3-agent was restarted.
    
    Add a check for these addresses and remove them at
    startup if they exist.  This can be removed in a cycle
    since once they are cleaned they will not be added
    back by the agent.
    
    Change-Id: Id512d213cd7ee11da913a4e4b0da20c3ad5420b0
    Closes-bug: #1675187


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1675187

Title:
  Floating IPs not removed on rfp interface in qrouter

Status in neutron:
  Fix Released

Bug description:
  Recently upgraded from liberty to newton. We had a lot of Active
  Floating IPs configured at the time.

  DVR setup, with 2 VMs on same network, and both with Floating IPs,
  FIPA and FIPB.

   - ssh into both FIPA and FIPB works from external source like laptop
   - SSH from one VM into another works via internal fixed IP ONLY(for example, ssh into floating of VMA, then ssh into fixed IP of VM B, or vice versa)
   - ping from one VM to floating IP of other *appears* to work. But even after deleting VM, pings continued. I suspect the rfp interface is responding to ICMP since it has FIP address configured

  Noticed that qrouter contained several /32 FIP addresses configured on
  rfp interface, but new Floating IPs we created were not being added as
  secondary IP addresses.

  Fix for bug https://bugs.launchpad.net/neutron/+bug/1462154 is
  responsible, I suspect. Note: this was backported into mitaka, so same
  issue should exist for liberty->mitaka upgrade.

  It removed the logic to both add and remove Floating IP on rfp - now
  the add/remove_floating_ip logic only handles the routes in the fip
  namespace, source ip rules in qrouter, etc.... The new agent is
  basically unaware of any floating IPs configured on the rfp interface.

  So it seems any pre-existing FIPs added as secondary IP address in
  qrouter remain as zombies. Attaching/reattahing, or deleting and re-
  creating does not remove these IPs as the logic to remove was also
  gone! Please see:
  https://review.openstack.org/#/c/289172/13/neutron/agent/l3/dvr_local_router.py

  As you can see below, lot of FIPs on 10.4.0.0/16 external network.
  Some correspond to VMs/floating IPs that were deleted. Others are
  still active but experiencing issue described above.

  Manually removing the IP immediately fixed our issue (ssh to floating
  IP from VM to VM worked immediately)

  root@barney:~# ip netns exec qrouter-37176403-cfb0-478d-b51c-971d89597cf5 ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
  2: rfp-37176403-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether 76:2f:be:73:9b:fc brd ff:ff:ff:ff:ff:ff
      inet 169.254.31.142/31 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.15/32 brd 10.4.253.15 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.118/32 brd 10.4.253.118 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.103/32 brd 10.4.252.103 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.105/32 brd 10.4.252.105 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.4/32 brd 10.4.254.4 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.52/32 brd 10.4.254.52 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.41/32 brd 10.4.253.41 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.228/32 brd 10.4.254.228 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.254.229/32 brd 10.4.254.229 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.253.212/32 brd 10.4.253.212 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.118/32 brd 10.4.252.118 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet 10.4.252.48/32 brd 10.4.252.48 scope global rfp-37176403-c
         valid_lft forever preferred_lft forever
      inet6 fe80::742f:beff:fe73:9bfc/64 scope link 
         valid_lft forever preferred_lft forever
  11: qr-43981e59-30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:42:a2:ec brd ff:ff:ff:ff:ff:ff
      inet 172.16.0.1/16 brd 172.16.255.255 scope global qr-43981e59-30
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe42:a2ec/64 scope link 
         valid_lft forever preferred_lft forever
  40083: qr-6b09eb9d-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:21:f8:fd brd ff:ff:ff:ff:ff:ff
      inet 192.168.42.100/24 brd 192.168.42.255 scope global qr-6b09eb9d-40
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe21:f8fd/64 scope link 
         valid_lft forever preferred_lft forever
  46229: qr-39850654-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
      link/ether fa:16:3e:f5:06:20 brd ff:ff:ff:ff:ff:ff
      inet 10.127.0.1/16 brd 10.127.255.255 scope global qr-39850654-40
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fef5:620/64 scope link 
         valid_lft forever preferred_lft forever

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1675187/+subscriptions