yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87222
[Bug 1930414] Re: Traffic leaked from dhcp port before vlan tag is applied
Thanks Bence! Based on your findings, the OpenStack VMT will consider
this a class C2 report (vulnerability in a dependency) per our taxonomy:
https://security.openstack.org/vmt-process.html#report-taxonomy
If new information comes to light, please update this bug and we'll
revisit the classification.
** Information type changed from Public Security to Public
** Changed in: ossa
Status: Incomplete => Won't Fix
** Tags added: security
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1930414
Title:
Traffic leaked from dhcp port before vlan tag is applied
Status in neutron:
New
Status in OpenStack Security Advisory:
Won't Fix
Bug description:
This is a bug with potential security implications. I don't see a
clear way to exploit it at the moment, but to err on the safe side,
I'm opening this as private to the security team.
Short summary: Using openvswitch-agent, traffic sent on some (at least
dhcp) ports before ovs-agent applies the port's vlan tag can be seen
and intercepted on ports from other networks on the same integration
bridge.
We observed this bug:
* using vlan and vxlan networks
* using the noop and openvswitch firewall drivers
* on openstack versions mitaka, pike and master (commit 5a6f61af4a)
The time window between the port's creation and ovs-agent applying its
vlan tag is usually very short. We observed this bug in the wild on a
heavily loaded host. However to make the reproduction reliable on
lightly loaded systems I inserted a sleep() into ovs-agent's source
(just before the port's vlan tag is set):
$ git --no-pager format-patch --stdout 5a6f61af4a
From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001
From: Bence Romsics <bence.romsics@xxxxxxxxx>
Date: Mon, 31 May 2021 13:12:34 +0200
Subject: [PATCH] WIP
Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb
---
.../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
index 2c209bd387..355584b325 100644
--- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
+++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
@@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin,
self.setup_arp_spoofing_protection(self.int_br,
port, port_detail)
if cur_tag != lvm.vlan:
+ time.sleep(3000)
self.int_br.set_db_attribute(
"Port", port.port_name, "tag", lvm.vlan)
·
--·
2.31.0
We discovered the bug by the following procedure:
* a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges
* therefore neutron selected the exact same ipv6 address for the subnet's dhcp port
* the host running the dhcp-agent and the ovs-agent was heavily loaded
* we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable
* because duplicate address detection (DAD) for the ipv6 dhcp port address failed
* we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks
* we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it
Exact reproduction steps:
$ date --iso-8601=s
2021-05-31T13:10:14+00:00
# when the ovs-agent is slow enough, even 2 networks are sufficient
$ for i in {1..5}
do
openstack network create xnet$i
openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32
done
# for the record
$ openstack subnet list -f value -c Name -c ID | egrep xsubnet
01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6
72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6
8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6
9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6
d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6
# note that all dhcp ports got the same ip
$ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8::
130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}]
19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}]
46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}]
b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}]
ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}]
# all but one dhcp port (and by the way metadata) addresses are in DAD failed mode
$ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))'
link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global dadfailed tentative·
inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative·
link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff
inet6 2001:db8::1/32 scope global·
inet6 fe80::a9fe:a9fe/64 scope link·
# dmesg also shows the DAD failures
# man dmesg: Be aware that the timestamp could be inaccurate!
$ LC_TIME=en_US sudo dmesg -T
[snip]
[Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode
[Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode
[Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode
[Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected!
[Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected!
# while there were no errors in ovs-agent
$ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR
[empty]
# clean up
$ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete
In a bit of further analysis I only created two networks and ran
tcpdump on the first's dhcp port, while creating the second:
$ openstack network create xnet0
$ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32
# run tcpdump on the 1st net's dhcp port
$ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53
# create the 2nd net while tcpdump is already running
$ openstack network create xnet1
$ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32
# the 2nd net's dhcp port's mac address
$ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address
fa:16:3e:0d:be:aa
# tcpdump's capture contains packets with the 2nd net's dhcp port's mac address
tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes
^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa
unknown option (14), length 8 (1):·
0x0000: bfdb db94 d695
14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1
unknown option (14), length 8 (1):·
0x0000: 6e67 6a99 e72e
14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe
unknown option (14), length 8 (1):·
0x0000: 11e1 9aab 157a
14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override]
destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb
14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)]
11 packets captured
11 packets received by filter
0 packets dropped by kernel
At the moment I did not test yet:
* whether the leaked traffic can be intercepted on a vm port
* whether a vm port can similarly leak traffic
* whether ovs-agent can be attacked to intentionally slow it down
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1930414/+subscriptions
