yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1493788] [NEW] DVR: Restarting the OVS agent does not re-create some of br-tun's flows

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: John Schwarz <jschwarz@xxxxxxxxxx>
Date: Wed, 09 Sep 2015 10:26:51 -0000
Reply-to: Bug 1493788 <1493788@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When, on a setup that has a DVR router, an OVS agent restarts, it fails
to re-create some of the flows for br-tun. For example:

$ # flows before agent restart
$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=77.325s, table=0, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
 cookie=0xa30fd64e48832cbc, duration=77.292s, table=0, n_packets=0, n_bytes=0, idle_age=190, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0xa30fd64e48832cbc, duration=77.281s, table=1, n_packets=0, n_bytes=0, idle_age=190, priority=0 actions=resubmit(,2)
 cookie=0x0, duration=77.324s, table=2, n_packets=0, n_bytes=0, idle_age=77, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=77.324s, table=2, n_packets=0, n_bytes=0, idle_age=77, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=77.324s, table=3, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
 cookie=0x0, duration=77.324s, table=4, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
 cookie=0xa30fd64e48832cbc, duration=77.018s, table=4, n_packets=0, n_bytes=0, idle_age=193, priority=1,tun_id=0x437 actions=mod_vlan_vid:1,resubmit(,9)
 cookie=0x0, duration=77.323s, table=6, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
 cookie=0xa30fd64e48832cbc, duration=77.286s, table=9, n_packets=0, n_bytes=0, idle_age=190, priority=0 actions=resubmit(,10)
 cookie=0xa30fd64e48832cbc, duration=77.259s, table=9, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_src=fa:16:3f:55:8c:22 actions=output:1
 cookie=0x0, duration=77.323s, table=10, n_packets=0, n_bytes=0, idle_age=77, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xa30fd64e48832cbc,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x0, duration=77.323s, table=20, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=resubmit(,22)
 cookie=0xa30fd64e48832cbc, duration=77.317s, table=22, n_packets=0, n_bytes=0, idle_age=193, priority=0 actions=drop

$ # flows after agent restart
$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0xbcfcd1c3b35d83e3, duration=3.072s, table=0, n_packets=0, n_bytes=0, idle_age=223, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0xbcfcd1c3b35d83e3, duration=3.060s, table=1, n_packets=0, n_bytes=0, idle_age=223, priority=0 actions=resubmit(,2)
 cookie=0xbcfcd1c3b35d83e3, duration=2.997s, table=4, n_packets=0, n_bytes=0, idle_age=226, priority=1,tun_id=0x437 actions=mod_vlan_vid:1,resubmit(,9)
 cookie=0xbcfcd1c3b35d83e3, duration=3.067s, table=9, n_packets=0, n_bytes=0, idle_age=223, priority=0 actions=resubmit(,10)
 cookie=0xbcfcd1c3b35d83e3, duration=3.038s, table=9, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_src=fa:16:3f:55:8c:22 actions=output:1
 cookie=0xbcfcd1c3b35d83e3, duration=3.100s, table=22, n_packets=0, n_bytes=0, idle_age=226, priority=0 actions=drop

It is clear that quite a few flows are missing. They can be re-created
by deleting all the flows on br-int - this starts a chain reaction which
ultimately recreates all the flows, including br-tun's.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1493788

Title:
  DVR: Restarting the OVS agent does not re-create some of br-tun's
  flows

Status in neutron:
  New

Bug description:
  When, on a setup that has a DVR router, an OVS agent restarts, it
  fails to re-create some of the flows for br-tun. For example:

  $ # flows before agent restart
  $ sudo ovs-ofctl dump-flows br-tun
  NXST_FLOW reply (xid=0x4):
   cookie=0x0, duration=77.325s, table=0, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
   cookie=0xa30fd64e48832cbc, duration=77.292s, table=0, n_packets=0, n_bytes=0, idle_age=190, priority=1,in_port=1 actions=resubmit(,1)
   cookie=0xa30fd64e48832cbc, duration=77.281s, table=1, n_packets=0, n_bytes=0, idle_age=190, priority=0 actions=resubmit(,2)
   cookie=0x0, duration=77.324s, table=2, n_packets=0, n_bytes=0, idle_age=77, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
   cookie=0x0, duration=77.324s, table=2, n_packets=0, n_bytes=0, idle_age=77, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
   cookie=0x0, duration=77.324s, table=3, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
   cookie=0x0, duration=77.324s, table=4, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
   cookie=0xa30fd64e48832cbc, duration=77.018s, table=4, n_packets=0, n_bytes=0, idle_age=193, priority=1,tun_id=0x437 actions=mod_vlan_vid:1,resubmit(,9)
   cookie=0x0, duration=77.323s, table=6, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=drop
   cookie=0xa30fd64e48832cbc, duration=77.286s, table=9, n_packets=0, n_bytes=0, idle_age=190, priority=0 actions=resubmit(,10)
   cookie=0xa30fd64e48832cbc, duration=77.259s, table=9, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_src=fa:16:3f:55:8c:22 actions=output:1
   cookie=0x0, duration=77.323s, table=10, n_packets=0, n_bytes=0, idle_age=77, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xa30fd64e48832cbc,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
   cookie=0x0, duration=77.323s, table=20, n_packets=0, n_bytes=0, idle_age=77, priority=0 actions=resubmit(,22)
   cookie=0xa30fd64e48832cbc, duration=77.317s, table=22, n_packets=0, n_bytes=0, idle_age=193, priority=0 actions=drop

  $ # flows after agent restart
  $ sudo ovs-ofctl dump-flows br-tun
  NXST_FLOW reply (xid=0x4):
   cookie=0xbcfcd1c3b35d83e3, duration=3.072s, table=0, n_packets=0, n_bytes=0, idle_age=223, priority=1,in_port=1 actions=resubmit(,1)
   cookie=0xbcfcd1c3b35d83e3, duration=3.060s, table=1, n_packets=0, n_bytes=0, idle_age=223, priority=0 actions=resubmit(,2)
   cookie=0xbcfcd1c3b35d83e3, duration=2.997s, table=4, n_packets=0, n_bytes=0, idle_age=226, priority=1,tun_id=0x437 actions=mod_vlan_vid:1,resubmit(,9)
   cookie=0xbcfcd1c3b35d83e3, duration=3.067s, table=9, n_packets=0, n_bytes=0, idle_age=223, priority=0 actions=resubmit(,10)
   cookie=0xbcfcd1c3b35d83e3, duration=3.038s, table=9, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_src=fa:16:3f:55:8c:22 actions=output:1
   cookie=0xbcfcd1c3b35d83e3, duration=3.100s, table=22, n_packets=0, n_bytes=0, idle_age=226, priority=0 actions=drop

  It is clear that quite a few flows are missing. They can be re-created
  by deleting all the flows on br-int - this starts a chain reaction
  which ultimately recreates all the flows, including br-tun's.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1493788/+subscriptions
Follow ups

[Bug 1493788] Re: DVR: Restarting the OVS agent does not re-create some of br-tun's flows
From: John Schwarz, 2015-09-09