yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88102
[Bug 1930414] Re: Traffic leaked from dhcp port before vlan tag is applied
Reviewed: https://review.opendev.org/c/openstack/neutron/+/820897
Committed: https://opendev.org/openstack/neutron/commit/7aae31c9f9ed938760ca0be3c461826b598c7004
Submitter: "Zuul (22348)"
Branch: master
commit 7aae31c9f9ed938760ca0be3c461826b598c7004
Author: Bence Romsics <bence.romsics@xxxxxxxxx>
Date: Tue Oct 5 17:02:41 2021 +0200
Make the dead vlan actually dead
All ports plugged into the dead vlan (DEAD_VLAN_TAG 4095 or 0xfff)
should not be able to send or receive traffic. We install a flow
to br-int to drop all traffic of the dead vlan [1]. However before
this patch the flow we install looks like:
priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
Which is wrong and it usually does not match anything.
According to ovs-fields (7) section Open vSwitch Extension VLAN Field,
VLAN TCI Field [2] (see especially the usage example
vlan_tci=0x1123/0x1fff) we need to explicitly set the bit 0x1000
to match the presence of an 802.1Q header.
Setting that bit this flow becomes:
priority=65535,vlan_tci=0x1fff/0x1fff actions=drop
which is equivalent to:
priority=65535,dl_vlan=4095 actions=drop
which should match and drop dead vlan traffic.
However there's a second problem: ovs access ports were designed to
work together with the NORMAL action. The NORMAL action considers the
vlan of an access port, but the openflow pipeline does not. An openflow
rule does not see the vlan set for an access port, because that vlan
tag is only pushed to the frame if and when the frame leaves the switch
on a trunk port [3][4].
So we have to explicitly push the DEAD_VLAN_TAG if we want the dead
vlan's drop flow match anything.
That means we are adding a flow to push the dead vlan tag from
dhcp-agent/l3-agent but we are deleting that flow from ovs-agent right
after ovs-agent sets the vlan tag of the port to a non-dead vlan. Which
is ugly but we have to keep adding the flow as early as possible if we
want to minimize the window until frames can leak onto the dead vlan.
Even with this change there's a short time window in which the dead vlan
could theoretically leak.
[1] https://opendev.org/openstack/neutron/src/commit/ecdc11a56448428f77f5a64fd028f1e4c9644ea3/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py#L60-L62
[2] http://www.openvswitch.org/support/dist-docs/ovs-fields.7.html
[3] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-December/051647.html
[4] https://docs.openvswitch.org/en/latest/faq/vlan/
see 'Q: My OpenFlow controller doesn’t see the VLANs that I expect.'
Change-Id: Ib6b70114efb140cf1393b57ebc350fea4b0a2443
Closes-Bug: #1930414
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1930414
Title:
Traffic leaked from dhcp port before vlan tag is applied
Status in neutron:
Fix Released
Status in OpenStack Security Advisory:
Won't Fix
Bug description:
This is a bug with potential security implications. I don't see a
clear way to exploit it at the moment, but to err on the safe side,
I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least
dhcp) ports before ovs-agent applies the port's vlan tag can be seen
and intercepted on ports from other networks on the same integration
bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its
vlan tag is usually very short. We observed this bug in the wild on a
heavily loaded host. However to make the reproduction reliable on
lightly loaded systems I inserted a sleep() into ovs-agent's source
(just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@xxxxxxxxx>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran
tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1930414/+subscriptions
