yahoo-eng-team team mailing list archive

[Nate Johnston <1838473@xxxxxxxxxxxxxxxxxx>]

*** This bug is a security vulnerability ***

Public security bug reported:

Background of the Issue
=======================
Customers expect that when they set Neutron security group rules such that all traffic is blocked that all traffic is in fact blocked by the Neutron firewall driver.  Neutron has multiple firewall drivers available, but in certain distributions (RHOSP for example) the default is the iptables_hybrid driver.

The iptables_hybrid driver was implemented using the iptables toolset.
This means that it is very effective at filtering IP traffic.  But IPv4
and IPv6 are only 2 of the available ethernet types (ethertypes).  There
are other types of traffic that ride over ethernet that are not IP [1],
and the iptables_hybrid firewall driver does not inspect or control
them.

First contact with this issue: A customer alerted us to an issue because
they were transitioning from the iptables_hybrid firewall driver to the
OpenVSwitch firewall driver (ovsfw).  When they made the switch they
noticed that all of their InfiniBand traffic was being blocked, because
it uses a non-IP ethertype (0x4008).  The fact that with iptables_hybrid
the traffic passed without any specific control enabling it is the
actual issue, we realized as we evaluated the case.

Analysis of the Vulnerability
=============================
Non-standard ethertypes are layer 3 protocols that are transmitted within the layer 2 ethernet frame.  This means that non-IP ethertypes cannot use IP-based routing to gain egress from the layer 2 domain.  When the iptables_hybrid firewall is in use:

- In clouds using self-service networking, instances on tenant networks depend on either Floating IPs or Neutron Routers for egress from their networks.  Since both of those are IP-specific concepts, traffic with custom ethertypes would have no native means to egress the network.
- In clouds using provider networking, instances depend on the provider network hardware to route traffic outside of the layer 2 domain.  Some ethertypes may be routable by network hardware - some routers understand InfiniBand or FCoE natively, for example, and will serve the routing implementation of those protocols.  We are not knowdgeable about the variety of routers out there and what they may support; it is entirely possible that legacy support for older protocols that have different ethertypes like IPX or PPPoE may exist in Cisco, Juniper, or other code..
- In ether case, if unauthorized access to a host is obtained by an intruder, then other hosts on the same L2 domain may be succeptible to the use of traffic travelling over other ethertypes to bypass hostbased firewalling. Exfiltration from that network from the compromised host is still subject to the restrictions in the above 2 bullet points.

To succinctly summarize the above: because non-IP based ethertypes
bypass all hostbased controls, only network-based controls restrict
them.  The best network-based control is the lack of support for routing
out of the L2 domain, which is mostly but not always true.

Proposed strategy
=================
We propose that a response to this issue has two parts.

1. The iptables_hybrid firewall is changed to deny ethertypes other than
IPv4, IPv6, and ARP by default.  Those 3 protocols are not covered
because they are handled through other mechanisms in the Neutron
security group system.  We propose that the iptables_hybrid control to
drop other ethertypes by default be enabled by default but configurable,
so that an operator could disable it either within the upgrade context
to maintain compatibility with conditions pre-upgrade, as an issue
triage step, or as part of a looser security policy.

2. The Neutron security group system must be enhanced with a control
that allows specific ethertypes to be permitted.  In upstream Neutron
master development this should be an addition to the existing Security
Groups API; for older Neutron versions this will be a configuration file
option.  This configuration file option should be manageable through
Director/TripleO.

This issue has been discussed in the upstream Neutron community [2] and
there is consensus on proposed points #1 and #2.

[1] https://en.wikipedia.org/wiki/EtherType#Examples
[2] http://eavesdrop.openstack.org/meetings/neutron_drivers/2019/neutron_drivers.2019-06-28-14.00.log.html

** Affects: neutron
     Importance: Undecided
     Assignee: Nate Johnston (nate-johnston)
         Status: Confirmed

** Description changed:

  Background of the Issue
  =======================
- Customers expect that when they set Neutron security group rules such that all
- traffic is blocked that all traffic is in fact blocked by the Neutron firewall
- driver.  Neutron has multiple firewall drivers available, but in certain 
- distributions (RHOSP for example) the default is the iptables_hybrid driver.
+ Customers expect that when they set Neutron security group rules such that all traffic is blocked that all traffic is in fact blocked by the Neutron firewall driver.  Neutron has multiple firewall drivers available, but in certain distributions (RHOSP for example) the default is the iptables_hybrid driver.
  
- The iptables_hybrid driver was implemented using the iptables toolset.  This
- means that it is very effective at filtering IP traffic.  But IPv4 and IPv6 are
- only 2 of the available ethernet types (ethertypes).  There are other types of
- traffic that ride over ethernet that are not IP [1], and the iptables_hybrid
- firewall driver does not inspect or control them.
+ The iptables_hybrid driver was implemented using the iptables toolset.
+ This means that it is very effective at filtering IP traffic.  But IPv4
+ and IPv6 are only 2 of the available ethernet types (ethertypes).  There
+ are other types of traffic that ride over ethernet that are not IP [1],
+ and the iptables_hybrid firewall driver does not inspect or control
+ them.
  
- First contact with this issue: A customer alerted us to an issue because they were
- transitioning from the iptables_hybrid firewall driver to the OpenVSwitch
- firewall driver (ovsfw).  When they made the switch they noticed that all of
- their InfiniBand traffic was being blocked, because it uses a non-IP ethertype
- (0x4008).  The fact that with iptables_hybrid the traffic passed without any
- specific control enabling it is the actual issue, we realized as we evaluated
- the case.
+ First contact with this issue: A customer alerted us to an issue because
+ they were transitioning from the iptables_hybrid firewall driver to the
+ OpenVSwitch firewall driver (ovsfw).  When they made the switch they
+ noticed that all of their InfiniBand traffic was being blocked, because
+ it uses a non-IP ethertype (0x4008).  The fact that with iptables_hybrid
+ the traffic passed without any specific control enabling it is the
+ actual issue, we realized as we evaluated the case.
  
  Analysis of the Vulnerability
  =============================
- Non-standard ethertypes are layer 3 protocols that are transmitted within the
- layer 2 ethernet frame.  This means that non-IP ethertypes cannot use IP-based
- routing to gain egress from the layer 2 domain.  When the iptables_hybrid
- firewall is in use:
+ Non-standard ethertypes are layer 3 protocols that are transmitted within the layer 2 ethernet frame.  This means that non-IP ethertypes cannot use IP-based routing to gain egress from the layer 2 domain.  When the iptables_hybrid firewall is in use:
  
- - In clouds using self-service networking, instances on tenant networks depend
-   on either Floating IPs or Neutron Routers for egress from their networks.
-   Since both of those are IP-specific concepts, traffic with custom ethertypes
-   would have no native means to egress the network.
- - In clouds using provider networking, instances depend on the provider network
-   hardware to route traffic outside of the layer 2 domain.  Some ethertypes may
-   be routable by network hardware - some routers understand InfiniBand or FCoE
-   natively, for example, and will serve the routing implementation of those
-   protocols.  We are not knowdgeable about the variety of routers out there and
-   what they may support; it is entirely possible that legacy support for older
-   protocols that have different ethertypes like IPX or PPPoE may exist in Cisco,
-   Juniper, or other code..
- - In ether case, if unauthorized access to a host is obtained by an intruder,
-   then other hosts on the same L2 domain may be succeptible to the use of
-   traffic travelling over other ethertypes to bypass hostbased firewalling.
-   Exfiltration from that network from the compromised host is still subject to the
-   restrictions in the above 2 bullet points.
+ - In clouds using self-service networking, instances on tenant networks depend on either Floating IPs or Neutron Routers for egress from their networks.  Since both of those are IP-specific concepts, traffic with custom ethertypes would have no native means to egress the network.
+ - In clouds using provider networking, instances depend on the provider network hardware to route traffic outside of the layer 2 domain.  Some ethertypes may be routable by network hardware - some routers understand InfiniBand or FCoE natively, for example, and will serve the routing implementation of those protocols.  We are not knowdgeable about the variety of routers out there and what they may support; it is entirely possible that legacy support for older protocols that have different ethertypes like IPX or PPPoE may exist in Cisco, Juniper, or other code..
+ - In ether case, if unauthorized access to a host is obtained by an intruder, then other hosts on the same L2 domain may be succeptible to the use of traffic travelling over other ethertypes to bypass hostbased firewalling. Exfiltration from that network from the compromised host is still subject to the restrictions in the above 2 bullet points.
  
- To succinctly summarize the above: because non-IP based ethertypes bypass all
- hostbased controls, only network-based controls restrict them.  The best
- network-based control is the lack of support for routing out of the L2 domain,
- which is mostly but not always true.
+ To succinctly summarize the above: because non-IP based ethertypes
+ bypass all hostbased controls, only network-based controls restrict
+ them.  The best network-based control is the lack of support for routing
+ out of the L2 domain, which is mostly but not always true.
  
  Proposed strategy
  =================
  We propose that a response to this issue has two parts.
  
- 1. The iptables_hybrid firewall is changed to deny ethertypes other than IPv4,
- IPv6, and ARP by default.  Those 3 protocols are not covered because they are
- handled through other mechanisms in the Neutron security group system.  We
- propose that the iptables_hybrid control to drop other ethertypes by default be
- enabled by default but configurable, so that an operator could disable it either
- within the upgrade context to maintain compatibility with conditions
- pre-upgrade, as an issue triage step, or as part of a looser security policy.
+ 1. The iptables_hybrid firewall is changed to deny ethertypes other than
+ IPv4, IPv6, and ARP by default.  Those 3 protocols are not covered
+ because they are handled through other mechanisms in the Neutron
+ security group system.  We propose that the iptables_hybrid control to
+ drop other ethertypes by default be enabled by default but configurable,
+ so that an operator could disable it either within the upgrade context
+ to maintain compatibility with conditions pre-upgrade, as an issue
+ triage step, or as part of a looser security policy.
  
- 2. The Neutron security group system must be enhanced with a control that allows
- specific ethertypes to be permitted.  In upstream Neutron master development
- this should be an addition to the existing Security Groups API; for older
- Neutron versions this will be a configuration file option.  This configuration
- file option should be manageable through Director/TripleO.
+ 2. The Neutron security group system must be enhanced with a control
+ that allows specific ethertypes to be permitted.  In upstream Neutron
+ master development this should be an addition to the existing Security
+ Groups API; for older Neutron versions this will be a configuration file
+ option.  This configuration file option should be manageable through
+ Director/TripleO.
  
- This issue has been discussed in the upstream Neutron community [2] and there is
- consensus on proposed points #1 and #2.
+ This issue has been discussed in the upstream Neutron community [2] and
+ there is consensus on proposed points #1 and #2.
  
  [1] https://en.wikipedia.org/wiki/EtherType#Examples
  [2] http://eavesdrop.openstack.org/meetings/neutron_drivers/2019/neutron_drivers.2019-06-28-14.00.log.html

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1838473

Title:
  non-IP ethertypes are permitted with iptables_hybrid firewall driver

Status in neutron:
  Confirmed

Bug description:
  Background of the Issue
  =======================
  Customers expect that when they set Neutron security group rules such that all traffic is blocked that all traffic is in fact blocked by the Neutron firewall driver.  Neutron has multiple firewall drivers available, but in certain distributions (RHOSP for example) the default is the iptables_hybrid driver.

  The iptables_hybrid driver was implemented using the iptables toolset.
  This means that it is very effective at filtering IP traffic.  But
  IPv4 and IPv6 are only 2 of the available ethernet types (ethertypes).
  There are other types of traffic that ride over ethernet that are not
  IP [1], and the iptables_hybrid firewall driver does not inspect or
  control them.

  First contact with this issue: A customer alerted us to an issue
  because they were transitioning from the iptables_hybrid firewall
  driver to the OpenVSwitch firewall driver (ovsfw).  When they made the
  switch they noticed that all of their InfiniBand traffic was being
  blocked, because it uses a non-IP ethertype (0x4008).  The fact that
  with iptables_hybrid the traffic passed without any specific control
  enabling it is the actual issue, we realized as we evaluated the case.

  Analysis of the Vulnerability
  =============================
  Non-standard ethertypes are layer 3 protocols that are transmitted within the layer 2 ethernet frame.  This means that non-IP ethertypes cannot use IP-based routing to gain egress from the layer 2 domain.  When the iptables_hybrid firewall is in use:

  - In clouds using self-service networking, instances on tenant networks depend on either Floating IPs or Neutron Routers for egress from their networks.  Since both of those are IP-specific concepts, traffic with custom ethertypes would have no native means to egress the network.
  - In clouds using provider networking, instances depend on the provider network hardware to route traffic outside of the layer 2 domain.  Some ethertypes may be routable by network hardware - some routers understand InfiniBand or FCoE natively, for example, and will serve the routing implementation of those protocols.  We are not knowdgeable about the variety of routers out there and what they may support; it is entirely possible that legacy support for older protocols that have different ethertypes like IPX or PPPoE may exist in Cisco, Juniper, or other code..
  - In ether case, if unauthorized access to a host is obtained by an intruder, then other hosts on the same L2 domain may be succeptible to the use of traffic travelling over other ethertypes to bypass hostbased firewalling. Exfiltration from that network from the compromised host is still subject to the restrictions in the above 2 bullet points.

  To succinctly summarize the above: because non-IP based ethertypes
  bypass all hostbased controls, only network-based controls restrict
  them.  The best network-based control is the lack of support for
  routing out of the L2 domain, which is mostly but not always true.

  Proposed strategy
  =================
  We propose that a response to this issue has two parts.

  1. The iptables_hybrid firewall is changed to deny ethertypes other
  than IPv4, IPv6, and ARP by default.  Those 3 protocols are not
  covered because they are handled through other mechanisms in the
  Neutron security group system.  We propose that the iptables_hybrid
  control to drop other ethertypes by default be enabled by default but
  configurable, so that an operator could disable it either within the
  upgrade context to maintain compatibility with conditions pre-upgrade,
  as an issue triage step, or as part of a looser security policy.

  2. The Neutron security group system must be enhanced with a control
  that allows specific ethertypes to be permitted.  In upstream Neutron
  master development this should be an addition to the existing Security
  Groups API; for older Neutron versions this will be a configuration
  file option.  This configuration file option should be manageable
  through Director/TripleO.

  This issue has been discussed in the upstream Neutron community [2]
  and there is consensus on proposed points #1 and #2.

  [1] https://en.wikipedia.org/wiki/EtherType#Examples
  [2] http://eavesdrop.openstack.org/meetings/neutron_drivers/2019/neutron_drivers.2019-06-28-14.00.log.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1838473/+subscriptions