yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88695
[Bug 1969615] [NEW] OVS: flow loop is created with openvswitch version 2.16
Public bug reported:
* Summary
neutron-openvswitch-agent is causing a flow loop when using Openvswitch version 2.16.
* High level description
Running Neutron in Xena release using the openvswitch plugin is causing a flow loop when using openvswitch version 2.16. This does not occur when deploying openvswitch version 2.15.
* Pre-conditions
Ansible-Kolla based deployment using "source: ubuntu" in stable/xena release. neutron_plugin_agent: "openvswitch". Deploying a 3 node cluster with basic Openstack services.
* Version:
** OpenStack version: Xena
** Linux distro: kolla-ansible stable/xena, Ubuntu 20.04.4 LTS
* Step-by-step
1. Deploy Openstack using kolla-ansible from stable/xena branch
2. Create a project network/subnet for Octavia
3. Create Octavia health-manager ports in Neutron for the 3 control nodes
4. Create the ports on each control node as ovs bridge ports
5. Assign IP addresses to the o-hm0 interfaces on all 3 nodes
6. try to ping one node from another node
ubuntu@ctl1:~$ openstack network show lb-mgmt
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2022-04-20T10:36:26Z |
| description | |
| dns_domain | None |
| id | c0c1b3ec-a6c3-4145-b94a-6c7fa4d7a740 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 1450 |
| name | lb-mgmt |
| port_security_enabled | True |
| project_id | 6cbb86e577a042499529110f6a1e8603 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 577 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | bf004f5a-4cae-4277-a3f4-a4cf787033cb |
| tags | |
| updated_at | 2022-04-20T10:36:28Z |
+---------------------------+--------------------------------------+
ubuntu@ctl1:~$ openstack subnet show lb-mgmt
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 172.16.1.1-172.16.255.254 |
| cidr | 172.16.0.0/16 |
| created_at | 2022-04-20T10:36:28Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 172.16.0.1 |
| host_routes | |
| id | bf004f5a-4cae-4277-a3f4-a4cf787033cb |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | lb-mgmt |
| network_id | c0c1b3ec-a6c3-4145-b94a-6c7fa4d7a740 |
| project_id | 6cbb86e577a042499529110f6a1e8603 |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2022-04-20T10:36:28Z |
+----------------------+--------------------------------------+
openstack port list --device-owner octavia:health-mgrr
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
| b0c8a28b-b652-4dce-a1b2-4a81e74d74ad | lb-mgmt-ctl1 | fa:17:20:16:00:11 | ip_address='172.16.0.11', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
| cfdb1171-21de-448b-a6b4-5c473e13ca12 | lb-mgmt-ctl3 | fa:17:20:16:00:13 | ip_address='172.16.0.13', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
| ea66498a-edb9-415b-b49e-fb005b635d75 | lb-mgmt-ctl2 | fa:17:20:16:00:12 | ip_address='172.16.0.12', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
ubuntu@ctl1:~$ ip a s o-hm0
10: o-hm0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:17:20:16:00:11 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.11/16 brd 172.16.255.255 scope global o-hm0
valid_lft forever preferred_lft forever
inet6 fe80::f817:20ff:fe16:11/64 scope link
valid_lft forever preferred_lft forever
ubuntu@ctl1:~$ ovs-vsctl --columns name,external-ids,ofport,error find Interface name='o-hm0'
name : o-hm0
external_ids : {attached-mac="fa:17:20:16:00:11", iface-id="b0c8a28b-b652-4dce-a1b2-4a81e74d74ad", iface-status=active, skip_cleanup="true"}
ofport : 4
error : []
ubuntu@ctl3:~$ ip a s o-hm0
9: o-hm0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:17:20:16:00:13 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.13/16 brd 172.16.255.255 scope global o-hm0
valid_lft forever preferred_lft forever
inet6 fe80::f817:20ff:fe16:13/64 scope link
valid_lft forever preferred_lft forever
ubuntu@ctl3:~$ ovs-vsctl --columns name,external-ids,ofport,error find Interface name='o-hm0'
name : o-hm0
external_ids : {attached-mac="fa:17:20:16:00:13", iface-id="cfdb1171-21de-448b-a6b4-5c473e13ca12", iface-status=active, skip_cleanup="true"}
ofport : 3
error : []
ubuntu@ctl1:~$ ping 172.16.0.13
PING 172.16.0.13 (172.16.0.13) 56(84) bytes of data.
^C
--- 172.16.0.13 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1025ms
ctl1:/var/log/kolla/openvswitch/ovs-vswitchd.log:
2022-04-20T10:38:14.051Z|00004|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:38:14.051Z|00005|ofproto_dpif_upcall(handler1)|WARN|Flow: ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x557db8f9194e8693
goto_table:60
60. priority 3, cookie 0x557db8f9194e8693
NORMAL
>>>> received packet on unknown port 6 <<<<
>> no input bundle, dropping
Final flow: unchanged
Megaflow: recirc_id=0,eth,ip,in_port=6,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_frag=no
Datapath actions: drop
2022-04-20T10:38:47.322Z|00092|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 10 flow_mods in the 2 s starting 53 s ago (10 adds)
2022-04-20T10:39:12.418Z|00006|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:57:38.785Z|00007|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:57:38.785Z|00008|ofproto_dpif_upcall(handler1)|WARN|Dropped 1 log messages in last 1107 seconds (most recently, 1107 seconds ago) due to excessive rate
2022-04-20T10:57:38.786Z|00009|ofproto_dpif_upcall(handler1)|WARN|Flow: ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x557db8f9194e8693
goto_table:60
60. priority 3, cookie 0x557db8f9194e8693
NORMAL
>>>> received packet on unknown port 6 <<<<
>> no input bundle, dropping
Final flow: unchanged
Megaflow: recirc_id=0,eth,ip,in_port=6,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_frag=no
Datapath actions: drop
** Affects: neutron
Importance: Undecided
Status: New
** Tags: ovs
** Tags added: ovs
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1969615
Title:
OVS: flow loop is created with openvswitch version 2.16
Status in neutron:
New
Bug description:
* Summary
neutron-openvswitch-agent is causing a flow loop when using Openvswitch version 2.16.
* High level description
Running Neutron in Xena release using the openvswitch plugin is causing a flow loop when using openvswitch version 2.16. This does not occur when deploying openvswitch version 2.15.
* Pre-conditions
Ansible-Kolla based deployment using "source: ubuntu" in stable/xena release. neutron_plugin_agent: "openvswitch". Deploying a 3 node cluster with basic Openstack services.
* Version:
** OpenStack version: Xena
** Linux distro: kolla-ansible stable/xena, Ubuntu 20.04.4 LTS
* Step-by-step
1. Deploy Openstack using kolla-ansible from stable/xena branch
2. Create a project network/subnet for Octavia
3. Create Octavia health-manager ports in Neutron for the 3 control nodes
4. Create the ports on each control node as ovs bridge ports
5. Assign IP addresses to the o-hm0 interfaces on all 3 nodes
6. try to ping one node from another node
ubuntu@ctl1:~$ openstack network show lb-mgmt
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2022-04-20T10:36:26Z |
| description | |
| dns_domain | None |
| id | c0c1b3ec-a6c3-4145-b94a-6c7fa4d7a740 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 1450 |
| name | lb-mgmt |
| port_security_enabled | True |
| project_id | 6cbb86e577a042499529110f6a1e8603 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 577 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | bf004f5a-4cae-4277-a3f4-a4cf787033cb |
| tags | |
| updated_at | 2022-04-20T10:36:28Z |
+---------------------------+--------------------------------------+
ubuntu@ctl1:~$ openstack subnet show lb-mgmt
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 172.16.1.1-172.16.255.254 |
| cidr | 172.16.0.0/16 |
| created_at | 2022-04-20T10:36:28Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 172.16.0.1 |
| host_routes | |
| id | bf004f5a-4cae-4277-a3f4-a4cf787033cb |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | lb-mgmt |
| network_id | c0c1b3ec-a6c3-4145-b94a-6c7fa4d7a740 |
| project_id | 6cbb86e577a042499529110f6a1e8603 |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2022-04-20T10:36:28Z |
+----------------------+--------------------------------------+
openstack port list --device-owner octavia:health-mgrr
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
| b0c8a28b-b652-4dce-a1b2-4a81e74d74ad | lb-mgmt-ctl1 | fa:17:20:16:00:11 | ip_address='172.16.0.11', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
| cfdb1171-21de-448b-a6b4-5c473e13ca12 | lb-mgmt-ctl3 | fa:17:20:16:00:13 | ip_address='172.16.0.13', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
| ea66498a-edb9-415b-b49e-fb005b635d75 | lb-mgmt-ctl2 | fa:17:20:16:00:12 | ip_address='172.16.0.12', subnet_id='bf004f5a-4cae-4277-a3f4-a4cf787033cb' | ACTIVE |
+--------------------------------------+--------------+-------------------+----------------------------------------------------------------------------+--------+
ubuntu@ctl1:~$ ip a s o-hm0
10: o-hm0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:17:20:16:00:11 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.11/16 brd 172.16.255.255 scope global o-hm0
valid_lft forever preferred_lft forever
inet6 fe80::f817:20ff:fe16:11/64 scope link
valid_lft forever preferred_lft forever
ubuntu@ctl1:~$ ovs-vsctl --columns name,external-ids,ofport,error find Interface name='o-hm0'
name : o-hm0
external_ids : {attached-mac="fa:17:20:16:00:11", iface-id="b0c8a28b-b652-4dce-a1b2-4a81e74d74ad", iface-status=active, skip_cleanup="true"}
ofport : 4
error : []
ubuntu@ctl3:~$ ip a s o-hm0
9: o-hm0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:17:20:16:00:13 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.13/16 brd 172.16.255.255 scope global o-hm0
valid_lft forever preferred_lft forever
inet6 fe80::f817:20ff:fe16:13/64 scope link
valid_lft forever preferred_lft forever
ubuntu@ctl3:~$ ovs-vsctl --columns name,external-ids,ofport,error find Interface name='o-hm0'
name : o-hm0
external_ids : {attached-mac="fa:17:20:16:00:13", iface-id="cfdb1171-21de-448b-a6b4-5c473e13ca12", iface-status=active, skip_cleanup="true"}
ofport : 3
error : []
ubuntu@ctl1:~$ ping 172.16.0.13
PING 172.16.0.13 (172.16.0.13) 56(84) bytes of data.
^C
--- 172.16.0.13 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1025ms
ctl1:/var/log/kolla/openvswitch/ovs-vswitchd.log:
2022-04-20T10:38:14.051Z|00004|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:38:14.051Z|00005|ofproto_dpif_upcall(handler1)|WARN|Flow: ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x557db8f9194e8693
goto_table:60
60. priority 3, cookie 0x557db8f9194e8693
NORMAL
>>>> received packet on unknown port 6 <<<<
>> no input bundle, dropping
Final flow: unchanged
Megaflow: recirc_id=0,eth,ip,in_port=6,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_frag=no
Datapath actions: drop
2022-04-20T10:38:47.322Z|00092|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 10 flow_mods in the 2 s starting 53 s ago (10 adds)
2022-04-20T10:39:12.418Z|00006|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:57:38.785Z|00007|ofproto_dpif_xlate(handler1)|WARN|over max translation depth 64 on bridge br-int while processing ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
2022-04-20T10:57:38.785Z|00008|ofproto_dpif_upcall(handler1)|WARN|Dropped 1 log messages in last 1107 seconds (most recently, 1107 seconds ago) due to excessive rate
2022-04-20T10:57:38.786Z|00009|ofproto_dpif_upcall(handler1)|WARN|Flow: ct_state=new|trk,ct_nw_src=172.16.0.11,ct_nw_dst=172.16.0.13,ct_nw_proto=1,ct_tp_src=8,ct_tp_dst=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_src=172.16.0.11,nw_dst=172.16.0.13,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x557db8f9194e8693
goto_table:60
60. priority 3, cookie 0x557db8f9194e8693
NORMAL
>>>> received packet on unknown port 6 <<<<
>> no input bundle, dropping
Final flow: unchanged
Megaflow: recirc_id=0,eth,ip,in_port=6,dl_src=fa:17:20:16:00:11,dl_dst=fa:17:20:16:00:13,nw_frag=no
Datapath actions: drop
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1969615/+subscriptions
Follow ups
