yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1912450] [NEW] flows on br-int wasn't been deleted completely

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: uchenily <1912450@xxxxxxxxxxxxxxxxxx>
Date: Wed, 20 Jan 2021 07:03:47 -0000
Reply-to: Bug 1912450 <1912450@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

hello guys, in our environments, we found that if we migrate(include
cold migration or live migration) / resize vms, there may be stale flows
left, we can't reproduce it 100%, but adding a sleep func before
'get_vifs_by_id'(https://opendev.org/openstack/neutron/src/tag/17.0.0/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1911-L1912)
will make it easier.

```
devices = devices_details_list.get('devices')
time.sleep(5) # make it easier to reproduce the problem
vif_by_id = self.int_br.get_vifs_by_ids(
```


steps to reproduce the problem:

1. create a vm on compute01 
openstack server create --image cirros --flavor m1.tiny --network vxlan9999 --wait --availability-zone :compute01 test20210120

2. get vm interface info, here we need 'c25976f2-ae' and mac address
nova interface-list test20210120
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| ACTIVE     | c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 | 5ea08661-ab6b-45dd-aba5-42346cb6ae70 | 172.16.0.25  | fa:16:3e:27:b5:63 |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+

3. dump-flows on compute01, we can grep the result by mac and in_port
ovs-ofctl dump-flows br-int --no-stats | grep -E "fa:16:3e:27:b5:63|in_port=3746"
 cookie=0x5b39cece80919832, priority=10,arp,in_port=3746 actions=resubmit(,24)
 cookie=0x5b39cece80919832, priority=10,icmp6,in_port=3746,icmp_type=136 actions=resubmit(,24)
 cookie=0x5b39cece80919832, priority=9,in_port=3746 actions=resubmit(,25)
 cookie=0x5b39cece80919832, table=1, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=mod_dl_src:fa:16:3e:73:64:50,resubmit(,60)
 cookie=0x5b39cece80919832, table=24, priority=2,icmp6,in_port=3746,icmp_type=136,nd_target=fe80::f816:3eff:fe27:b563 actions=resubmit(,60)
 cookie=0x5b39cece80919832, table=24, priority=2,arp,in_port=3746,arp_spa=172.16.0.25 actions=resubmit(,25)
 cookie=0x5b39cece80919832, table=25, priority=2,in_port=3746,dl_src=fa:16:3e:27:b5:63 actions=resubmit(,60)
 cookie=0x5b39cece80919832, table=60, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
 cookie=0x5b39cece80919832, table=60, priority=9,in_port=3746,dl_src=fa:16:3e:27:b5:63 actions=load:0x27->NXM_NX_REG6[],resubmit(,61)
 cookie=0x5b39cece80919832, table=61, priority=12,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
 cookie=0x5b39cece80919832, table=61, priority=12,reg6=0x27,dl_dst=fa:16:3e:27:b5:63 actions=output:3746
 cookie=0x5b39cece80919832, table=62, priority=10,in_port=3746,dl_src=fa:16:3e:27:b5:63,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:39,output:3

4. migrate this vm, you can use live-migration or cold-migration
openstack server migrate --live-migration --host compute02 test20210120 --wait --os-compute-api-version 2.30

5. after the migration, dump flows again
ovs-ofctl dump-flows br-int --no-stats | grep -E "fa:16:3e:27:b5:63|in_port=3746"
 cookie=0x5b39cece80919832, table=1, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=mod_dl_src:fa:16:3e:73:64:50,resubmit(,60)
 cookie=0x5b39cece80919832, table=60, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
 cookie=0x5b39cece80919832, table=61, priority=12,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
 cookie=0x5b39cece80919832, table=61, priority=12,reg6=0x27,dl_dst=fa:16:3e:27:b5:63 actions=output:3746

here we found table=1, 60, 61 has stale flows, the other flows has been
deleted because these flows match in_port:3746 and this devices has been
deleted from br-int; you may need to repeat several times to reproduce
it.


# here is ovs-agent logs, i have deleted unimportant ports informations in 'current' list.
2021-01-20 06:13:54.292 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Starting to process devices in:{'added': set(), 'removed': set(), 'current': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'aaa6f513-5de3-4be5-a468-d49e3b15856f'}, 'updated': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'}} rpc_loop /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2424
2021-01-20 06:13:59.329 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Processing port: c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 treat_devices_added_or_updated /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1693
2021-01-20 06:13:59.329 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 updated. Details: {'device': 'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'network_id': '5ea08661-ab6b-45dd-aba5-42346cb6ae70', 'port_id': 'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'mac_address': 'fa:16:3e:27:b5:63', 'admin_state_up': True, 'network_type': 'vxlan', 'segmentation_id': 9999, 'physical_network': None, 'fixed_ips': [{'subnet_id': 'fd3edeff-604b-462e-a0e0-d4a3c24ee7cc', 'ip_address': '172.16.0.25'}], 'device_owner': 'compute:cn-north-3a', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {'migrating_to': 'compute03'}, 'security_groups': ['0a13b7db-62fa-4dd0-ab03-071ab1967bfe']}
2021-01-20 06:13:59.330 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] DVR: Plumbing compute port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 _bind_port_on_dvr_subnet /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py:472
2021-01-20 06:14:01.477 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Setting status for c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 to UP _bind_devices /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:967
2021-01-20 06:14:02.404 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Configuration for devices up ['c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'] and devices down [] completed.
2021-01-20 06:14:02.448 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Starting to process devices in:{'added': set(), 'removed': set(), 'current': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'aaa6f513-5de3-4be5-a468-d49e3b15856f'}, 'updated': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'}} rpc_loop /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2424
2021-01-20 06:14:07.485 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Processing port: c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 treat_devices_added_or_updated /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1693
2021-01-20 06:14:07.486 1 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 was not found on the integration bridge and will therefore not be processed
2021-01-20 06:14:08.933 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Ports {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'} skipped, changing status to down

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1912450

Title:
  flows on br-int wasn't been deleted completely

Status in neutron:
  New

Bug description:
  hello guys, in our environments, we found that if we migrate(include
  cold migration or live migration) / resize vms, there may be stale
  flows left, we can't reproduce it 100%, but adding a sleep func before
  'get_vifs_by_id'(https://opendev.org/openstack/neutron/src/tag/17.0.0/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1911-L1912)
  will make it easier.

  ```
  devices = devices_details_list.get('devices')
  time.sleep(5) # make it easier to reproduce the problem
  vif_by_id = self.int_br.get_vifs_by_ids(
  ```

  
  steps to reproduce the problem:

  1. create a vm on compute01 
  openstack server create --image cirros --flavor m1.tiny --network vxlan9999 --wait --availability-zone :compute01 test20210120

  2. get vm interface info, here we need 'c25976f2-ae' and mac address
  nova interface-list test20210120
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+
  | Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          |
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+
  | ACTIVE     | c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 | 5ea08661-ab6b-45dd-aba5-42346cb6ae70 | 172.16.0.25  | fa:16:3e:27:b5:63 |
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+

  3. dump-flows on compute01, we can grep the result by mac and in_port
  ovs-ofctl dump-flows br-int --no-stats | grep -E "fa:16:3e:27:b5:63|in_port=3746"
   cookie=0x5b39cece80919832, priority=10,arp,in_port=3746 actions=resubmit(,24)
   cookie=0x5b39cece80919832, priority=10,icmp6,in_port=3746,icmp_type=136 actions=resubmit(,24)
   cookie=0x5b39cece80919832, priority=9,in_port=3746 actions=resubmit(,25)
   cookie=0x5b39cece80919832, table=1, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=mod_dl_src:fa:16:3e:73:64:50,resubmit(,60)
   cookie=0x5b39cece80919832, table=24, priority=2,icmp6,in_port=3746,icmp_type=136,nd_target=fe80::f816:3eff:fe27:b563 actions=resubmit(,60)
   cookie=0x5b39cece80919832, table=24, priority=2,arp,in_port=3746,arp_spa=172.16.0.25 actions=resubmit(,25)
   cookie=0x5b39cece80919832, table=25, priority=2,in_port=3746,dl_src=fa:16:3e:27:b5:63 actions=resubmit(,60)
   cookie=0x5b39cece80919832, table=60, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
   cookie=0x5b39cece80919832, table=60, priority=9,in_port=3746,dl_src=fa:16:3e:27:b5:63 actions=load:0x27->NXM_NX_REG6[],resubmit(,61)
   cookie=0x5b39cece80919832, table=61, priority=12,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
   cookie=0x5b39cece80919832, table=61, priority=12,reg6=0x27,dl_dst=fa:16:3e:27:b5:63 actions=output:3746
   cookie=0x5b39cece80919832, table=62, priority=10,in_port=3746,dl_src=fa:16:3e:27:b5:63,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:39,output:3

  4. migrate this vm, you can use live-migration or cold-migration
  openstack server migrate --live-migration --host compute02 test20210120 --wait --os-compute-api-version 2.30

  5. after the migration, dump flows again
  ovs-ofctl dump-flows br-int --no-stats | grep -E "fa:16:3e:27:b5:63|in_port=3746"
   cookie=0x5b39cece80919832, table=1, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=mod_dl_src:fa:16:3e:73:64:50,resubmit(,60)
   cookie=0x5b39cece80919832, table=60, priority=20,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
   cookie=0x5b39cece80919832, table=61, priority=12,dl_vlan=39,dl_dst=fa:16:3e:27:b5:63 actions=strip_vlan,output:3746
   cookie=0x5b39cece80919832, table=61, priority=12,reg6=0x27,dl_dst=fa:16:3e:27:b5:63 actions=output:3746

  here we found table=1, 60, 61 has stale flows, the other flows has
  been deleted because these flows match in_port:3746 and this devices
  has been deleted from br-int; you may need to repeat several times to
  reproduce it.

  
  # here is ovs-agent logs, i have deleted unimportant ports informations in 'current' list.
  2021-01-20 06:13:54.292 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Starting to process devices in:{'added': set(), 'removed': set(), 'current': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'aaa6f513-5de3-4be5-a468-d49e3b15856f'}, 'updated': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'}} rpc_loop /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2424
  2021-01-20 06:13:59.329 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Processing port: c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 treat_devices_added_or_updated /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1693
  2021-01-20 06:13:59.329 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 updated. Details: {'device': 'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'network_id': '5ea08661-ab6b-45dd-aba5-42346cb6ae70', 'port_id': 'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'mac_address': 'fa:16:3e:27:b5:63', 'admin_state_up': True, 'network_type': 'vxlan', 'segmentation_id': 9999, 'physical_network': None, 'fixed_ips': [{'subnet_id': 'fd3edeff-604b-462e-a0e0-d4a3c24ee7cc', 'ip_address': '172.16.0.25'}], 'device_owner': 'compute:cn-north-3a', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {'migrating_to': 'compute03'}, 'security_groups': ['0a13b7db-62fa-4dd0-ab03-071ab1967bfe']}
  2021-01-20 06:13:59.330 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] DVR: Plumbing compute port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 _bind_port_on_dvr_subnet /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py:472
  2021-01-20 06:14:01.477 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Setting status for c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 to UP _bind_devices /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:967
  2021-01-20 06:14:02.404 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Configuration for devices up ['c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'] and devices down [] completed.
  2021-01-20 06:14:02.448 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Starting to process devices in:{'added': set(), 'removed': set(), 'current': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3', 'aaa6f513-5de3-4be5-a468-d49e3b15856f'}, 'updated': {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'}} rpc_loop /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2424
  2021-01-20 06:14:07.485 1 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Processing port: c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 treat_devices_added_or_updated /var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1693
  2021-01-20 06:14:07.486 1 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Port c25976f2-aea4-4b78-9a8f-e4ec2b1373e3 was not found on the integration bridge and will therefore not be processed
  2021-01-20 06:14:08.933 1 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-6e6dde50-1636-45ec-a8ef-df1e15744209 - - - - -] Ports {'c25976f2-aea4-4b78-9a8f-e4ec2b1373e3'} skipped, changing status to down

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1912450/+subscriptions