yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #90203
[Bug 1995872] [NEW] A stuck INACTIVE port binding causes wrong l2pop fdb entries to be sent
Public bug reported:
We are testing the network availability of VMs in case of HA events.
And we run into a problem where aborting live migration of a VM can break communication with that VM in the future at the OVS rules level.
The fault of the wrong OVS rules is the stuck INACTIVE port binding in the neutron `ml2_port_bindings` table.
Steps to reproduce:
Install cluster via devstack with target branch `master` with 3 nodes.
mechanism driver is `openvswitch` with `l2population` enabled:
[root@node0 ~]# grep -r l2population /etc/neutron/*
/etc/neutron/plugins/ml2/ml2_conf.ini:mechanism_drivers = openvswitch,l2population
[root@node0 ~]#
0) preparation:
- create a vxlan based internal network,
- start 3 VMs per each node: vm0 -> node0, vm1 -> node1, vm2 -> node2
[root@node0 ~]# for i in {0..2}; do openstack server create vm$i --network vxlan-net --flavor m1.tiny --image cirros-0.5.2-x86_64-disk; done
[root@node0 ~]# for i in {0..2}; do openstack server migrate vm$i --host node$i --live-migration; done
1) abort the `vm1` live migration from node1 -> node0
[root@node0 ~]# openstack server migrate vm1 --host node0 --live-migration; sleep 1; ssh root@node1 systemctl stop devstack@n-cpu.service
[root@node0 ~]# openstack server list
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
| 56ec7007-5470-42df-863e-8ae7d6a0110f | vm1 | MIGRATING | vxlan-net=192.168.0.169 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 5bc93710-8da8-4b12-b1f0-767cf1768d27 | vm2 | ACTIVE | vxlan-net=192.168.0.82 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 6f93f40f-0065-413c-81e6-724a21b3756b | vm0 | ACTIVE | vxlan-net=192.168.0.135 | cirros-0.5.2-x86_64-disk | m1.tiny |
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
[root@node0 ~]# ssh root@node1 systemctl start devstack@n-cpu.service
[root@node0 ~]# openstack server list
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
| 56ec7007-5470-42df-863e-8ae7d6a0110f | vm1 | ACTIVE | vxlan-net=192.168.0.169 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 5bc93710-8da8-4b12-b1f0-767cf1768d27 | vm2 | ACTIVE | vxlan-net=192.168.0.82 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 6f93f40f-0065-413c-81e6-724a21b3756b | vm0 | ACTIVE | vxlan-net=192.168.0.135 | cirros-0.5.2-x86_64-disk | m1.tiny |
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
[root@node0 ~]#
VM failed to migrate and still on the node1:
[root@node0 ~]# openstack server show vm1 -c OS-EXT-SRV-ATTR:host
+----------------------+---------+
| Field | Value |
+----------------------+---------+
| OS-EXT-SRV-ATTR:host | node1 |
+----------------------+---------+
[root@node0 ~]# ssh node1 virsh list
Id Name State
-----------------------------------
3 instance-00000009 running
[root@node0 ~]#
Now I get two port bindings ACTIVE and INACTIVE for the `vm1` port:
MariaDB [neutron]> select port_id,host,vif_type,profile from ml2_port_bindings where port_id='3be55a45-83c6-42b7-82fc-fb6c4855f255';
+--------------------------------------+---------+----------+-----------------------------+
| port_id | host | vif_type | profile |
+--------------------------------------+---------+----------+-----------------------------+
| 3be55a45-83c6-42b7-82fc-fb6c4855f255 | node0 | ovs | {"os_vif_delegation": true} |
| 3be55a45-83c6-42b7-82fc-fb6c4855f255 | node1 | ovs | {"migrating_to": "node0"} |
+--------------------------------------+---------+----------+-----------------------------+
2) restart on the node2 the neutron-openvswitch-agent, that forces
neutron-server to repopulate neighbors fdb entries:
[root@node0 ~]# ssh node2 systemctl restart devstack@q-agt.service
Now a ping from the vm2 to the vm1 doesn't work:
[root@node0 ~]# ip netns exec qdhcp-f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2 ssh cirros@192.168.0.82
sign_and_send_pubkey: no mutual signature supported
cirros@192.168.0.82's password:
$ ping 192.168.0.169
PING 192.168.0.169 (192.168.0.169): 56 data bytes
^C
--- 192.168.0.169 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
$
This is because the br-tun rules on node2 send traffic for `vm1` to
node0 and not to node1 where VM is actually located:
[root@node2 ~]# ovs-appctl ofproto/trace br-int in_port=tapa6a78630-a4,ip,dl_src=fa:16:3e:08:9a:4f,dl_dst=fa:16:3e:9f:26:18,nw_src=192.168.0.82,nw_dst=192.168.0.169
...
bridge("br-int")
...
bridge("br-tun")
----------------
0. in_port=1, priority 1, cookie 0x5777965a7facb9aa
goto_table:1
1. priority 0, cookie 0x5777965a7facb9aa
goto_table:2
2. dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie 0x5777965a7facb9aa
goto_table:20
20. dl_vlan=2,dl_dst=fa:16:3e:9f:26:18, priority 2, cookie 0x5777965a7facb9aa
pop_vlan
set_field:0x339->tun_id
output:5
-> output to kernel tunnel
[root@node2 ~]# ovs-ofctl show br-tun | grep -w 5
5(vxlan-0a8810ba): addr:f2:78:a4:0d:2c:2c
[root@node2 ~]# ovs-vsctl show | grep -A 4 vxlan-0a8810ba
Port vxlan-0a8810ba
Interface vxlan-0a8810ba
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.136.16.78", out_key=flow, remote_ip="10.136.16.186"}
[root@node2 ~]#
Where IP remote_ip="10.136.16.186" is address of the node0.
I added a trace log to see what the neutron-server sends to the agent
after restarting the ovs-agent:
[root@node0 neutron]# git diff
diff --git a/neutron/plugins/ml2/drivers/l2pop/mech_driver.py b/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
index 80eb1a9bd0..523833f6c4 100644
--- a/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
+++ b/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
@@ -311,6 +311,7 @@ class L2populationMechanismDriver(api.MechanismDriver):
other_fdb_ports[agent_ip].append(const.FLOODING_ENTRY)
if agent_fdb_entries[network_id]['ports'].keys():
+ LOG.info('TRACE: sending agent_fdb_entries=%s', agent_fdb_entries)
self.L2populationAgentNotify.add_fdb_entries(
self.rpc_ctx, agent_fdb_entries, agent_host)
[root@node0 neutron]#
[root@node0 ~]# journalctl -f -u devstack@q-svc.service | grep -w TRACE
Nov 07 07:37:02 node0 neutron-server[102976]: INFO neutron.plugins.ml2.drivers.l2pop.mech_driver [None req-b77086e1-90e1-41bb-81a5-1c45d792fcf3 None None] TRACE: sending agent_fdb_entries={'f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2': {'segment_id': 825, 'network_type': 'vxlan', 'ports': {'10.136.19.188': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')], '10.136.16.186': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169'), PortInfo(mac_address='fa:16:3e:86:e4:cf', ip_address='192.168.0.135'), PortInfo(mac_address='fa:16:3e:98:f6:a8', ip_address='192.168.0.2')]}}}
Nov 07 07:37:04 node0 neutron-server[102976]: INFO neutron.plugins.ml2.drivers.l2pop.mech_driver [None req-b77086e1-90e1-41bb-81a5-1c45d792fcf3 None None] TRACE: sending agent_fdb_entries={'f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2': {'segment_id': 825, 'network_type': 'vxlan', 'ports': {'10.136.19.188': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')], '10.136.16.186': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169'), PortInfo(mac_address='fa:16:3e:86:e4:cf', ip_address='192.168.0.135'), PortInfo(mac_address='fa:16:3e:98:f6:a8', ip_address='192.168.0.2')]}}}
^C
[root@node0 ~]#
So,
'ports': {
'10.136.19.188': [..., PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')],
'10.136.16.186': [..., PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')]
}
The neutron-server sends that IP address 192.168.0.169 lives on both agents 10.136.19.188(node0) and 10.136.16.186(node1)
And now it depends on the neutron-ovs-agent in what order it apply these fdb entries.
So, it looks to me better to filter out the INACTIVE port bindings from the fdb entries population:
diff --git a/neutron/plugins/ml2/drivers/l2pop/db.py b/neutron/plugins/ml2/drivers/l2pop/db.py
index 38c22ac4f1..3bef2a3327 100644
--- a/neutron/plugins/ml2/drivers/l2pop/db.py
+++ b/neutron/plugins/ml2/drivers/l2pop/db.py
@@ -82,6 +82,7 @@ def _get_active_network_ports(context, network_id):
query = query.options(orm.subqueryload(ml2_models.PortBinding.port))
query = query.filter(models_v2.Port.network_id == network_id,
models_v2.Port.status == const.PORT_STATUS_ACTIVE)
+ query = query.filter(ml2_models.PortBinding.status == 'ACTIVE')
return query
Also, it seems there is also a question to the nova, why it does not clear the created port bindings.
But due to the fact that the nova service can be turned off indefinitely, it doesn't seem ok to keep l2pop broken on the neutron side.
If you can confirm the bug, I can prepare a patch.
Thanks in advance
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1995872
Title:
A stuck INACTIVE port binding causes wrong l2pop fdb entries to be
sent
Status in neutron:
New
Bug description:
We are testing the network availability of VMs in case of HA events.
And we run into a problem where aborting live migration of a VM can break communication with that VM in the future at the OVS rules level.
The fault of the wrong OVS rules is the stuck INACTIVE port binding in the neutron `ml2_port_bindings` table.
Steps to reproduce:
Install cluster via devstack with target branch `master` with 3 nodes.
mechanism driver is `openvswitch` with `l2population` enabled:
[root@node0 ~]# grep -r l2population /etc/neutron/*
/etc/neutron/plugins/ml2/ml2_conf.ini:mechanism_drivers = openvswitch,l2population
[root@node0 ~]#
0) preparation:
- create a vxlan based internal network,
- start 3 VMs per each node: vm0 -> node0, vm1 -> node1, vm2 -> node2
[root@node0 ~]# for i in {0..2}; do openstack server create vm$i --network vxlan-net --flavor m1.tiny --image cirros-0.5.2-x86_64-disk; done
[root@node0 ~]# for i in {0..2}; do openstack server migrate vm$i --host node$i --live-migration; done
1) abort the `vm1` live migration from node1 -> node0
[root@node0 ~]# openstack server migrate vm1 --host node0 --live-migration; sleep 1; ssh root@node1 systemctl stop devstack@n-cpu.service
[root@node0 ~]# openstack server list
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
| 56ec7007-5470-42df-863e-8ae7d6a0110f | vm1 | MIGRATING | vxlan-net=192.168.0.169 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 5bc93710-8da8-4b12-b1f0-767cf1768d27 | vm2 | ACTIVE | vxlan-net=192.168.0.82 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 6f93f40f-0065-413c-81e6-724a21b3756b | vm0 | ACTIVE | vxlan-net=192.168.0.135 | cirros-0.5.2-x86_64-disk | m1.tiny |
+--------------------------------------+------+-----------+-------------------------+--------------------------+---------+
[root@node0 ~]# ssh root@node1 systemctl start devstack@n-cpu.service
[root@node0 ~]# openstack server list
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
| 56ec7007-5470-42df-863e-8ae7d6a0110f | vm1 | ACTIVE | vxlan-net=192.168.0.169 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 5bc93710-8da8-4b12-b1f0-767cf1768d27 | vm2 | ACTIVE | vxlan-net=192.168.0.82 | cirros-0.5.2-x86_64-disk | m1.tiny |
| 6f93f40f-0065-413c-81e6-724a21b3756b | vm0 | ACTIVE | vxlan-net=192.168.0.135 | cirros-0.5.2-x86_64-disk | m1.tiny |
+--------------------------------------+------+--------+-------------------------+--------------------------+---------+
[root@node0 ~]#
VM failed to migrate and still on the node1:
[root@node0 ~]# openstack server show vm1 -c OS-EXT-SRV-ATTR:host
+----------------------+---------+
| Field | Value |
+----------------------+---------+
| OS-EXT-SRV-ATTR:host | node1 |
+----------------------+---------+
[root@node0 ~]# ssh node1 virsh list
Id Name State
-----------------------------------
3 instance-00000009 running
[root@node0 ~]#
Now I get two port bindings ACTIVE and INACTIVE for the `vm1` port:
MariaDB [neutron]> select port_id,host,vif_type,profile from ml2_port_bindings where port_id='3be55a45-83c6-42b7-82fc-fb6c4855f255';
+--------------------------------------+---------+----------+-----------------------------+
| port_id | host | vif_type | profile |
+--------------------------------------+---------+----------+-----------------------------+
| 3be55a45-83c6-42b7-82fc-fb6c4855f255 | node0 | ovs | {"os_vif_delegation": true} |
| 3be55a45-83c6-42b7-82fc-fb6c4855f255 | node1 | ovs | {"migrating_to": "node0"} |
+--------------------------------------+---------+----------+-----------------------------+
2) restart on the node2 the neutron-openvswitch-agent, that forces
neutron-server to repopulate neighbors fdb entries:
[root@node0 ~]# ssh node2 systemctl restart devstack@q-agt.service
Now a ping from the vm2 to the vm1 doesn't work:
[root@node0 ~]# ip netns exec qdhcp-f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2 ssh cirros@192.168.0.82
sign_and_send_pubkey: no mutual signature supported
cirros@192.168.0.82's password:
$ ping 192.168.0.169
PING 192.168.0.169 (192.168.0.169): 56 data bytes
^C
--- 192.168.0.169 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
$
This is because the br-tun rules on node2 send traffic for `vm1` to
node0 and not to node1 where VM is actually located:
[root@node2 ~]# ovs-appctl ofproto/trace br-int in_port=tapa6a78630-a4,ip,dl_src=fa:16:3e:08:9a:4f,dl_dst=fa:16:3e:9f:26:18,nw_src=192.168.0.82,nw_dst=192.168.0.169
...
bridge("br-int")
...
bridge("br-tun")
----------------
0. in_port=1, priority 1, cookie 0x5777965a7facb9aa
goto_table:1
1. priority 0, cookie 0x5777965a7facb9aa
goto_table:2
2. dl_dst=00:00:00:00:00:00/01:00:00:00:00:00, priority 0, cookie 0x5777965a7facb9aa
goto_table:20
20. dl_vlan=2,dl_dst=fa:16:3e:9f:26:18, priority 2, cookie 0x5777965a7facb9aa
pop_vlan
set_field:0x339->tun_id
output:5
-> output to kernel tunnel
[root@node2 ~]# ovs-ofctl show br-tun | grep -w 5
5(vxlan-0a8810ba): addr:f2:78:a4:0d:2c:2c
[root@node2 ~]# ovs-vsctl show | grep -A 4 vxlan-0a8810ba
Port vxlan-0a8810ba
Interface vxlan-0a8810ba
type: vxlan
options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.136.16.78", out_key=flow, remote_ip="10.136.16.186"}
[root@node2 ~]#
Where IP remote_ip="10.136.16.186" is address of the node0.
I added a trace log to see what the neutron-server sends to the agent
after restarting the ovs-agent:
[root@node0 neutron]# git diff
diff --git a/neutron/plugins/ml2/drivers/l2pop/mech_driver.py b/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
index 80eb1a9bd0..523833f6c4 100644
--- a/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
+++ b/neutron/plugins/ml2/drivers/l2pop/mech_driver.py
@@ -311,6 +311,7 @@ class L2populationMechanismDriver(api.MechanismDriver):
other_fdb_ports[agent_ip].append(const.FLOODING_ENTRY)
if agent_fdb_entries[network_id]['ports'].keys():
+ LOG.info('TRACE: sending agent_fdb_entries=%s', agent_fdb_entries)
self.L2populationAgentNotify.add_fdb_entries(
self.rpc_ctx, agent_fdb_entries, agent_host)
[root@node0 neutron]#
[root@node0 ~]# journalctl -f -u devstack@q-svc.service | grep -w TRACE
Nov 07 07:37:02 node0 neutron-server[102976]: INFO neutron.plugins.ml2.drivers.l2pop.mech_driver [None req-b77086e1-90e1-41bb-81a5-1c45d792fcf3 None None] TRACE: sending agent_fdb_entries={'f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2': {'segment_id': 825, 'network_type': 'vxlan', 'ports': {'10.136.19.188': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')], '10.136.16.186': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169'), PortInfo(mac_address='fa:16:3e:86:e4:cf', ip_address='192.168.0.135'), PortInfo(mac_address='fa:16:3e:98:f6:a8', ip_address='192.168.0.2')]}}}
Nov 07 07:37:04 node0 neutron-server[102976]: INFO neutron.plugins.ml2.drivers.l2pop.mech_driver [None req-b77086e1-90e1-41bb-81a5-1c45d792fcf3 None None] TRACE: sending agent_fdb_entries={'f0f8f0b6-3cd3-4ae5-b5cf-25f2834bcdb2': {'segment_id': 825, 'network_type': 'vxlan', 'ports': {'10.136.19.188': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')], '10.136.16.186': [('00:00:00:00:00:00', '0.0.0.0'), PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169'), PortInfo(mac_address='fa:16:3e:86:e4:cf', ip_address='192.168.0.135'), PortInfo(mac_address='fa:16:3e:98:f6:a8', ip_address='192.168.0.2')]}}}
^C
[root@node0 ~]#
So,
'ports': {
'10.136.19.188': [..., PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')],
'10.136.16.186': [..., PortInfo(mac_address='fa:16:3e:9f:26:18', ip_address='192.168.0.169')]
}
The neutron-server sends that IP address 192.168.0.169 lives on both agents 10.136.19.188(node0) and 10.136.16.186(node1)
And now it depends on the neutron-ovs-agent in what order it apply these fdb entries.
So, it looks to me better to filter out the INACTIVE port bindings from the fdb entries population:
diff --git a/neutron/plugins/ml2/drivers/l2pop/db.py b/neutron/plugins/ml2/drivers/l2pop/db.py
index 38c22ac4f1..3bef2a3327 100644
--- a/neutron/plugins/ml2/drivers/l2pop/db.py
+++ b/neutron/plugins/ml2/drivers/l2pop/db.py
@@ -82,6 +82,7 @@ def _get_active_network_ports(context, network_id):
query = query.options(orm.subqueryload(ml2_models.PortBinding.port))
query = query.filter(models_v2.Port.network_id == network_id,
models_v2.Port.status == const.PORT_STATUS_ACTIVE)
+ query = query.filter(ml2_models.PortBinding.status == 'ACTIVE')
return query
Also, it seems there is also a question to the nova, why it does not clear the created port bindings.
But due to the fact that the nova service can be turned off indefinitely, it doesn't seem ok to keep l2pop broken on the neutron side.
If you can confirm the bug, I can prepare a patch.
Thanks in advance
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1995872/+subscriptions
