← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #94653

[Bug 2083214] [NEW] [RFE] control random-fully behavior on a per-FIP base

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

As of this moment, Neutron uses random-fully[1] PAT when performing NAT on L3 agent, meaning that source port is randomized for every outgoing connection. This breaks some workflows for software that performs UDP hole punching, such as Nebula[2] for example. There're other use cases where knowing post-NAT source port value for an outgoing connection is desirable.
Neutron currently provides a `use_random_fully` setting[3] that controls use of random-fully PAT, but it's global and affects cloud as a whole.

My proposal is to implement control over random-fully setting on a per-
Floating-IP basis.

I've already implemented this in a Devstack environment. My change required updates in neutron(L3 agent code, database schema update(1 additional column of a boolean type for floatingip table)), neutron-lib(API support, introduction of a new validator type, etc), and openstackclient(CLI support).
In this implementation a new 'random_fully' setting belonging to a FIP can take 3 values: True, False, or None.
If True(API JSON: {"floatingip": {"random_fully": true}}), random-fully is always enabled on a FIP, disregarding global `use_random_fully` setting.
If False(API JSON: {"floatingip": {"random_fully": false}}), random-fully is always disabled on a FIP, disregarding global `use_random_fully` setting.
If None(API JSON: {"floatingip": {"random_fully": null}}), random-fully mode is inherited from the global `use_random_fully` setting.

It works pretty much as expected, L3 agent updates iptables rules after API call.
I'll be glad to share that code to expedite this feature implementation.

Short example output from a Devstack environment:
```
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c 'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Router                               | Status | Description          | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11         | 10.0.0.21        | b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE | My FIP description 4 | None         |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207        | 10.0.0.39        | 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | None         |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231        | 10.0.0.19        | 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep "neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
stack@vlab007:~/neutron$ openstack floating ip set --disable-random-fully 387fdc61-d386-4917-bd82-23055ebca273
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c 'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Router                               | Status | Description          | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11         | 10.0.0.21        | b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE | My FIP description 4 | None         |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207        | 10.0.0.39        | 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231        | 10.0.0.19        | 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep "neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
```

[1] https://ipset.netfilter.org/iptables-extensions.man.html
[2] https://github.com/slackhq/nebula
[3] https://opendev.org/openstack/neutron/commit/553777c52d0fbfa18a1830d41ac018d2e1dc8282

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2083214

Title:
  [RFE] control random-fully behavior on a per-FIP base

Status in neutron:
  New

Bug description:
  As of this moment, Neutron uses random-fully[1] PAT when performing NAT on L3 agent, meaning that source port is randomized for every outgoing connection. This breaks some workflows for software that performs UDP hole punching, such as Nebula[2] for example. There're other use cases where knowing post-NAT source port value for an outgoing connection is desirable.
  Neutron currently provides a `use_random_fully` setting[3] that controls use of random-fully PAT, but it's global and affects cloud as a whole.

  My proposal is to implement control over random-fully setting on a
  per-Floating-IP basis.

  I've already implemented this in a Devstack environment. My change required updates in neutron(L3 agent code, database schema update(1 additional column of a boolean type for floatingip table)), neutron-lib(API support, introduction of a new validator type, etc), and openstackclient(CLI support).
  In this implementation a new 'random_fully' setting belonging to a FIP can take 3 values: True, False, or None.
  If True(API JSON: {"floatingip": {"random_fully": true}}), random-fully is always enabled on a FIP, disregarding global `use_random_fully` setting.
  If False(API JSON: {"floatingip": {"random_fully": false}}), random-fully is always disabled on a FIP, disregarding global `use_random_fully` setting.
  If None(API JSON: {"floatingip": {"random_fully": null}}), random-fully mode is inherited from the global `use_random_fully` setting.

  It works pretty much as expected, L3 agent updates iptables rules after API call.
  I'll be glad to share that code to expedite this feature implementation.

  Short example output from a Devstack environment:
  ```
  stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c 'Random Fully'
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  | ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Router                               | Status | Description          | Random Fully |
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  | 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11         | 10.0.0.21        | b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE | My FIP description 4 | None         |
  | 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207        | 10.0.0.39        | 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | None         |
  | b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231        | 10.0.0.19        | 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  ubuntu@vlab007:~$ sudo ip netns exec qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep "neutron-l3-agent-float-snat -s"
  -A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11 --random-fully
  -A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207 --random-fully
  -A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
  stack@vlab007:~/neutron$ openstack floating ip set --disable-random-fully 387fdc61-d386-4917-bd82-23055ebca273
  stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c 'Random Fully'
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  | ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Router                               | Status | Description          | Random Fully |
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  | 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11         | 10.0.0.21        | b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE | My FIP description 4 | None         |
  | 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207        | 10.0.0.39        | 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
  | b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231        | 10.0.0.19        | 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 | ACTIVE |                      | False        |
  +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
  ubuntu@vlab007:~$ sudo ip netns exec qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep "neutron-l3-agent-float-snat -s"
  -A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11 --random-fully
  -A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207
  -A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
  ```

  [1] https://ipset.netfilter.org/iptables-extensions.man.html
  [2] https://github.com/slackhq/nebula
  [3] https://opendev.org/openstack/neutron/commit/553777c52d0fbfa18a1830d41ac018d2e1dc8282

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2083214/+subscriptions
  Thread Previous • Date Previous • • Thread Next