yahoo-eng-team team mailing list archive

[Michal Nasiadka <1922923@xxxxxxxxxxxxxxxxxx>]

** No longer affects: kolla-ansible

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1922923

Title:
  [L3] arp issue in router namespace in compute node

Status in neutron:
  New

Bug description:
  Hello,

  I have an issue with Openstack Victoria. Since I moved to Openstack
  Victoria I still have an issue. Sometimes the VM cannot be accessed on
  the FIP and it gets fixed only after I clear the arp table for the
  private IP of the VM from the network namespace.

  I did some troubleshooting and I found out that the port seems to down
  in the OVS.

  
   7(qr-4affa6db-67): addr:00:00:00:00:00:00
       config:     PORT_DOWN
       state:      LINK_DOWN
       speed: 0 Mbps now, 0 Mbps max

  
  The MAC address of the VM port is this one:

  [root@compute-38 ~]# arp -a | grep 87
  ? (10.10.13.87) at fa:16:3e:ee:d1:57 [ether] PERM on qr-4affa6db-67

  And the ping it's now working.

  [root@compute-38 ~]# ip r
  10.10.13.0/24 dev qr-4affa6db-67 proto kernel scope link src 10.10.13.1
  169.254.107.94/31 dev rfp-9b2225f1-b proto kernel scope link src 169.254.107.94
  169.254.110.46/31 dev rfp-9b2225f1-b proto kernel scope link src 169.254.110.46
  [root@compute-38 ~]# ping 10.10.13.1
  PING 10.10.13.1 (10.10.13.1) 56(84) bytes of data.
  64 bytes from 10.10.13.1: icmp_seq=1 ttl=64 time=0.050 ms
  ^C
  --- 10.10.13.1 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms
  [root@compute-38 ~]# ping 10.10.13.87
  PING 10.10.13.87 (10.10.13.87) 56(84) bytes of data.
  ^C
  --- 10.10.13.87 ping statistics ---
  3 packets transmitted, 0 received, 100% packet loss, time 87ms

  [root@compute-38 ~]#

  
  The solution is to clear the ARP for 10.10.13.87 from the namespace.

  
  [root@compute-38 ~]# ping 10.10.13.87
  PING 10.10.13.87 (10.10.13.87) 56(84) bytes of data.
  ^C
  --- 10.10.13.87 ping statistics ---
  3 packets transmitted, 0 received, 100% packet loss, time 87ms

  [root@compute-38 ~]# arp -d 10.10.13.87
  [root@compute-38 ~]# arp -a | grep 87
  ? (10.10.13.87) at fa:16:3e:99:08:a5 [ether] on qr-4affa6db-67
  [root@compute-38 ~]# ping 10.10.13.87
  PING 10.10.13.87 (10.10.13.87) 56(84) bytes of data.
  64 bytes from 10.10.13.87: icmp_seq=1 ttl=64 time=0.322 ms
  64 bytes from 10.10.13.87: icmp_seq=2 ttl=64 time=0.239 ms
  ^C
  --- 10.10.13.87 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 12ms
  rtt min/avg/max/mdev = 0.239/0.280/0.322/0.044 ms

  And now the FIP started to answer:

  ~]#  ping 10.40.131.220
  PING 10.40.131.220 (10.40.131.220) 56(84) bytes of data.
  64 bytes from 10.40.131.220: icmp_seq=2500 ttl=61 time=1.60 ms
  64 bytes from 10.40.131.220: icmp_seq=2501 ttl=61 time=0.462 ms
  64 bytes from 10.40.131.220: icmp_seq=2502 ttl=61 time=0.536 ms

  ^C
  --- 10.40.131.220 ping statistics ---
  2545 packets transmitted, 46 received, 98% packet loss, time 2544013ms
  rtt min/avg/max/mdev = 0.305/0.571/1.606/0.272 ms

  And it's weird that after it's working the ports still looks down.

  7(qr-4affa6db-67): addr:00:00:00:00:00:00
       config:     PORT_DOWN
       state:      LINK_DOWN
       speed: 0 Mbps now, 0 Mbps max

  From what I see it seems like a bug as it gets fixed after I do this
  workaround. And this is happening to the brand new VMs that are being
  deployed, but not all of them.

  Do you have any idea how can I fix this issue? I updated the
  containers last week to the latest stable release of Victoria.

  Thanks!
  Andrei

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1922923/+subscriptions