yahoo-eng-team team mailing list archive

[Florian Haas <2072505@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When deciding whether a floating IP can be assigned to a port, the check
on whether to accept the API call, or return HTTP 400 with an "External
network not reachable from subnet" error message, does not take static
routes on a router into account — when arguably it should, or at least
could.

This applies mainly to situations where users want to use a Nova server
(for example, a virtual security appliance) as a firewall, router, or
gateway for other servers.

To illustrate, suppose a regular user is dealing with

* an external network,
* an "outer" internal network,
* a Neutron router connecting the two (that is, the router has the external network as its gateway, and has an interface to the "outer" subnet),
* an "inner" internal network,
* a Nova server (let's call that the "gateway"), which connects the  "inner" and "outer" networks,
* another Nova server (the "worker") that sits on the "inner" network.

(Suppose further, for the sake of discussion, that the "gateway" runs
some kind of web application firewall or other virtual security
appliance, and the "worker" is a general-purpose Linux instance.)

Now one can give the Neutron router a static route so that traffic to
the "inner" network uses the "gateway" as its nexthop. Provided the
"gateway" does its forwarding correctly, this means that the "worker"
has internet connectivity via the "gateway" (which, in turn, achieves
this via the router).

This, too, works fine for outbound connections originating with the
"worker".

If one now wants to assign a floating IP to the "worker", then routing-
wise there really should be no problem: the router should take care of
SNAT/DNAT, it would translate the floating IP into a private IP on the
"inner" network, would consult its routing table, and send packets for
that "inner" network to the "gateway" to handle, per the static route
that's been set.

But when one tries to associate a floating IP to the "worker" in this
scenario, one gets the "External network not reachable from subnet"
error, because the associated internal check[1] does not take the
existing static route into account.

This appears to have been an issue in other contexts as well: the
defunct Tricircle project appears to have worked around this limitation
via a service plugin that just omitted that check.[2]

One of the questions that were raised in discussing this on the list was
how Neutron could determine that the "inner" network is indeed
accessible through the "gateway" VM. This is considering that from
Neutron's perspective, it's just a VM with two legs in each of the two
networks, and forwarding inside the "gateway" VM is opaque.

However, from the router's perspective, that should not really matter.

Currently, simplifying in very broad strokes, the router checks its own
interfaces_info map to determine if it is connected to a specific
subnet. It then checks whether that subnet's netmask matches the private
IP of the floating/fixed IP association.

Neutron can't really check if that private IP address is actually
"accessible": a user could shell into that server and disable the
interface, and the router would be none the wiser. It would just keep
sending packets the server's way, and they would time out.

Now, if the router, when determining whether it can host a floating IP,
were to also check its own routes map to see if

* the private IP (on the "inner" network) matched a route destination, and
* the route nexthop matched a subnet that the router was directly
connected to (namely, the "outer" network),

it could be just as sufficient a plausibility check for routing
purposes.

After all, what users expect of the router is to create DNAT/SNAT rules
for the floating IP addresses, and with said information from routes
this should be possible, with as much plausibility checking as for
directly connected subnets.

So I guess the feature enhancement would boil down to this:

1. Make Neutron routers consult their own static routes when determining whether they are eligible for hosting a floating IP.
2. Possibly add a configuration flag to be able to toggle this on and off.

References:
[1] get_router_for_floatingip():
https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42add6eee571663528/neutron/db/l3_db.py#L1270

[2] get_router_for_floatingip():
https://opendev.org/openstack/tricircle/src/commit/8db8fb30f5757c46a953962d8c2133743cfffdb3/tricircle/network/local_l3_plugin.py#L30

Original openstack-discuss thread:
https://lists.openstack.org/archives/list/openstack-
discuss@xxxxxxxxxxxxxxxxxxx/thread/7OINJKMEV5P2KDKJCEYH672EQUKX57A3/

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: rfe

** Description changed:

  When deciding whether a floating IP can be assigned to a port, the check
  on whether to accept the API call, or return HTTP 400 with an "External
  network not reachable from subnet" error message, does not take static
- routes on a router into account — when arguably it should.
+ routes on a router into account — when arguably it should, or at least
+ could.
  
- 
- This applies mainly to situations where users want to use a Nova server (for example, a virtual security appliance) as a firewall, router, or gateway for other servers.
- 
+ This applies mainly to situations where users want to use a Nova server
+ (for example, a virtual security appliance) as a firewall, router, or
+ gateway for other servers.
  
  To illustrate, suppose a regular user is dealing with
  
  * an external network,
  * an "outer" internal network,
  * a Neutron router connecting the two (that is, the router has the external network as its gateway, and has an interface to the "outer" subnet),
  * an "inner" internal network,
- * a Nova server (let's call that the "gateway"), which connects the  "inner" and "outer" networks, 
+ * a Nova server (let's call that the "gateway"), which connects the  "inner" and "outer" networks,
  * another Nova server (the "worker") that sits on the "inner" network.
  
  (Suppose further, for the sake of discussion, that the "gateway" runs
  some kind of web application firewall or other virtual security
  appliance, and the "worker" is a general-purpose Linux instance.)
  
  Now one can give the Neutron router a static route so that traffic to
  the "inner" network uses the "gateway" as its nexthop. Provided the
  "gateway" does its forwarding correctly, this means that the "worker"
  has internet connectivity via the "gateway" (which, in turn, achieves
  this via the router).
  
  This, too, works fine for outbound connections originating with the
  "worker".
  
  If one now wants to assign a floating IP to the "worker", then routing-
  wise there really should be no problem: the router should take care of
  SNAT/DNAT, it would translate the floating IP into a private IP on the
  "inner" network, would consult its routing table, and send packets for
  that "inner" network to the "gateway" to handle, per the static route
  that's been set.
  
  But when one tries to associate a floating IP to the "worker" in this
  scenario, one gets the "External network not reachable from subnet"
  error, because the associated internal check[1] does not take the
  existing static route into account.
  
  This appears to have been an issue in other contexts as well: the
  defunct Tricircle project appears to have worked around this limitation
  via a service plugin that just omitted that check.[2]
  
- 
- One of the questions that were raised in discussing this on the list was how Neutron could determine that the "inner" network is indeed accessible through the "gateway" VM. This is considering that from Neutron's perspective, it's just a VM with two legs in each of the two networks, and forwarding inside the "gateway" VM is opaque.
- 
+ One of the questions that were raised in discussing this on the list was
+ how Neutron could determine that the "inner" network is indeed
+ accessible through the "gateway" VM. This is considering that from
+ Neutron's perspective, it's just a VM with two legs in each of the two
+ networks, and forwarding inside the "gateway" VM is opaque.
  
  However, from the router's perspective, that should not really matter.
  
  Currently, simplifying in very broad strokes, the router checks its own
  interfaces_info map to determine if it is connected to a specific
  subnet. It then checks whether that subnet's netmask matches the private
  IP of the floating/fixed IP association.
  
  Neutron can't really check if that private IP address is actually
  "accessible": a user could shell into that server and disable the
  interface, and the router would be none the wiser. It would just keep
  sending packets the server's way, and they would time out.
  
  Now, if the router, when determining whether it can host a floating IP,
  were to also check its own routes map to see if
  
  * the private IP (on the "inner" network) matched a route destination, and
- * the route nexthop matched a subnet that the router was directly 
+ * the route nexthop matched a subnet that the router was directly
  connected to (namely, the "outer" network),
  
  it could be just as sufficient a plausibility check for routing
  purposes.
  
  After all, what users expect of the router is to create DNAT/SNAT rules
  for the floating IP addresses, and with said information from routes
  this should be possible, with as much plausibility checking as for
  directly connected subnets.
  
- 
  So I guess the feature enhancement would boil down to this:
  
  1. Make Neutron routers consult their own static routes when determining whether they are eligible for hosting a floating IP.
  2. Possibly add a configuration flag to be able to toggle this on and off.
  
+ References:
+ [1] get_router_for_floatingip():
+ https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42add6eee571663528/neutron/db/l3_db.py#L1270
  
- References:
- [1] get_router_for_floatingip(): 
- https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42add6eee571663528/neutron/db/l3_db.py#L1270 
- 
- [2] get_router_for_floatingip(): 
+ [2] get_router_for_floatingip():
  https://opendev.org/openstack/tricircle/src/commit/8db8fb30f5757c46a953962d8c2133743cfffdb3/tricircle/network/local_l3_plugin.py#L30
  
- 
- Original openstack-discuss thread: https://lists.openstack.org/archives/list/openstack-discuss@xxxxxxxxxxxxxxxxxxx/thread/7OINJKMEV5P2KDKJCEYH672EQUKX57A3/
+ Original openstack-discuss thread:
+ https://lists.openstack.org/archives/list/openstack-
+ discuss@xxxxxxxxxxxxxxxxxxx/thread/7OINJKMEV5P2KDKJCEYH672EQUKX57A3/

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2072505

Title:
  "External network not reachable from subnet" could take static routes
  into account

Status in neutron:
  New

Bug description:
  When deciding whether a floating IP can be assigned to a port, the
  check on whether to accept the API call, or return HTTP 400 with an
  "External network not reachable from subnet" error message, does not
  take static routes on a router into account — when arguably it should,
  or at least could.

  This applies mainly to situations where users want to use a Nova
  server (for example, a virtual security appliance) as a firewall,
  router, or gateway for other servers.

  To illustrate, suppose a regular user is dealing with

  * an external network,
  * an "outer" internal network,
  * a Neutron router connecting the two (that is, the router has the external network as its gateway, and has an interface to the "outer" subnet),
  * an "inner" internal network,
  * a Nova server (let's call that the "gateway"), which connects the  "inner" and "outer" networks,
  * another Nova server (the "worker") that sits on the "inner" network.

  (Suppose further, for the sake of discussion, that the "gateway" runs
  some kind of web application firewall or other virtual security
  appliance, and the "worker" is a general-purpose Linux instance.)

  Now one can give the Neutron router a static route so that traffic to
  the "inner" network uses the "gateway" as its nexthop. Provided the
  "gateway" does its forwarding correctly, this means that the "worker"
  has internet connectivity via the "gateway" (which, in turn, achieves
  this via the router).

  This, too, works fine for outbound connections originating with the
  "worker".

  If one now wants to assign a floating IP to the "worker", then
  routing-wise there really should be no problem: the router should take
  care of SNAT/DNAT, it would translate the floating IP into a private
  IP on the "inner" network, would consult its routing table, and send
  packets for that "inner" network to the "gateway" to handle, per the
  static route that's been set.

  But when one tries to associate a floating IP to the "worker" in this
  scenario, one gets the "External network not reachable from subnet"
  error, because the associated internal check[1] does not take the
  existing static route into account.

  This appears to have been an issue in other contexts as well: the
  defunct Tricircle project appears to have worked around this
  limitation via a service plugin that just omitted that check.[2]

  One of the questions that were raised in discussing this on the list
  was how Neutron could determine that the "inner" network is indeed
  accessible through the "gateway" VM. This is considering that from
  Neutron's perspective, it's just a VM with two legs in each of the two
  networks, and forwarding inside the "gateway" VM is opaque.

  However, from the router's perspective, that should not really matter.

  Currently, simplifying in very broad strokes, the router checks its
  own interfaces_info map to determine if it is connected to a specific
  subnet. It then checks whether that subnet's netmask matches the
  private IP of the floating/fixed IP association.

  Neutron can't really check if that private IP address is actually
  "accessible": a user could shell into that server and disable the
  interface, and the router would be none the wiser. It would just keep
  sending packets the server's way, and they would time out.

  Now, if the router, when determining whether it can host a floating
  IP, were to also check its own routes map to see if

  * the private IP (on the "inner" network) matched a route destination, and
  * the route nexthop matched a subnet that the router was directly
  connected to (namely, the "outer" network),

  it could be just as sufficient a plausibility check for routing
  purposes.

  After all, what users expect of the router is to create DNAT/SNAT
  rules for the floating IP addresses, and with said information from
  routes this should be possible, with as much plausibility checking as
  for directly connected subnets.

  So I guess the feature enhancement would boil down to this:

  1. Make Neutron routers consult their own static routes when determining whether they are eligible for hosting a floating IP.
  2. Possibly add a configuration flag to be able to toggle this on and off.

  References:
  [1] get_router_for_floatingip():
  https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42add6eee571663528/neutron/db/l3_db.py#L1270

  [2] get_router_for_floatingip():
  https://opendev.org/openstack/tricircle/src/commit/8db8fb30f5757c46a953962d8c2133743cfffdb3/tricircle/network/local_l3_plugin.py#L30

  Original openstack-discuss thread:
  https://lists.openstack.org/archives/list/openstack-
  discuss@xxxxxxxxxxxxxxxxxxx/thread/7OINJKMEV5P2KDKJCEYH672EQUKX57A3/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2072505/+subscriptions