← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #79508

[Bug 1839004] [NEW] Rocky DVR-SNAT seems missing entries for conntrack marking

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Hello,

I've been playing with Rocky on CentOS 7 with DVR / DVR-SNAT / BGP. Our
provider network uses private IP-space for scalability. Our tenants run
in public IP-space.


Steps:

openstack network create --share --provider-network-type vlan --external --provider-physical-network EXTERNAL NET-EXT-VLAN625-UPLINK --provider-segment 625
openstack subnet create --network NET-EXT-VLAN625-UPLINK --subnet-range 192.0.2.0/24 SUBNET-EXT-VLAN625-UPLINK
openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-EXT-VLAN625-UPLINK

openstack address scope create --share --ip-version 4 SCOPE-SHARED-VFOUR
openstack subnet pool create --pool-prefix 93.115.169.128/25 --address-scope SCOPE-SHARED-VFOUR POOL-SHARED-EXTERNAL
openstack subnet pool set --default-prefix-length 28 --min-prefix-length 26 POOL-SHARED-EXTERNAL

openstack network create --share NET-INT-PUBLIC-SHARED
openstack subnet create --network NET-INT-PUBLIC-SHARED --subnet-pool POOL-SHARED-EXTERNAL --prefix-length 28 SUBNET-INT-PUBLIC-SHARED
openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-INT-PUBLIC-SHARED

openstack router create ROUTER-SHARED
openstack router set ROUTER-SHARED --disable-snat --external-gateway NET-EXT-VLAN625-UPLINK
openstack router add subnet ROUTER-SHARED SUBNET-INT-PUBLIC-SHARED
openstack bgp speaker list advertised routes BGP-REFLECTOR-SHARED

Result:

+-------------------+-----------+
| Destination       | Nexthop   |
+-------------------+-----------+
| 93.115.169.128/28 | 192.0.2.6 |
+-------------------+-----------+

Where 192.0.2.6 is the IP in the SNAT instance on the DVR-SNAT network
node. So far things seem good however it seems my egress traffic is
blocked. I'm not 100% sure it's not caused by misconfiguration but I
found the two lines to get traffic flowing in my config:

https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1093
was rewritten to "dont_block_external = (ip_version == lib_constants.IP_VERSION_4 and external_port)" 

-> This would allow traffic to go egress into the provider network

https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1122
if-return statement was commented out.

-> I need that connmark/mark restore rule to have ingress replies match
the existing connections.

Validate:

ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t mangle -L neutron-l3-agent-POSTROUTING -nv
Chain neutron-l3-agent-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
508 27174 CONNMARK all -- * qg-b075d908-66 0.0.0.0/0 0.0.0.0/0 connmark match 0x0/0xffff0000 CONNMARK save mask 0xffff0000


1 entry

ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t filter -L neutron-l3-agent-scope -nv
Chain neutron-l3-agent-scope (1 references)
pkts bytes target prot opt in out source destination
6588 350K DROP all -- * sg-c46c9df8-06 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4010000/0xffff0000

1 entry (instead of two before)

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1839004

Title:
  Rocky DVR-SNAT seems missing entries for conntrack marking

Status in neutron:
  New

Bug description:
  Hello,

  I've been playing with Rocky on CentOS 7 with DVR / DVR-SNAT / BGP.
  Our provider network uses private IP-space for scalability. Our
  tenants run in public IP-space.

  
  Steps:

  openstack network create --share --provider-network-type vlan --external --provider-physical-network EXTERNAL NET-EXT-VLAN625-UPLINK --provider-segment 625
  openstack subnet create --network NET-EXT-VLAN625-UPLINK --subnet-range 192.0.2.0/24 SUBNET-EXT-VLAN625-UPLINK
  openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-EXT-VLAN625-UPLINK

  openstack address scope create --share --ip-version 4 SCOPE-SHARED-VFOUR
  openstack subnet pool create --pool-prefix 93.115.169.128/25 --address-scope SCOPE-SHARED-VFOUR POOL-SHARED-EXTERNAL
  openstack subnet pool set --default-prefix-length 28 --min-prefix-length 26 POOL-SHARED-EXTERNAL

  openstack network create --share NET-INT-PUBLIC-SHARED
  openstack subnet create --network NET-INT-PUBLIC-SHARED --subnet-pool POOL-SHARED-EXTERNAL --prefix-length 28 SUBNET-INT-PUBLIC-SHARED
  openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-INT-PUBLIC-SHARED

  openstack router create ROUTER-SHARED
  openstack router set ROUTER-SHARED --disable-snat --external-gateway NET-EXT-VLAN625-UPLINK
  openstack router add subnet ROUTER-SHARED SUBNET-INT-PUBLIC-SHARED
  openstack bgp speaker list advertised routes BGP-REFLECTOR-SHARED

  Result:

  +-------------------+-----------+
  | Destination       | Nexthop   |
  +-------------------+-----------+
  | 93.115.169.128/28 | 192.0.2.6 |
  +-------------------+-----------+

  Where 192.0.2.6 is the IP in the SNAT instance on the DVR-SNAT network
  node. So far things seem good however it seems my egress traffic is
  blocked. I'm not 100% sure it's not caused by misconfiguration but I
  found the two lines to get traffic flowing in my config:

  https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1093
  was rewritten to "dont_block_external = (ip_version == lib_constants.IP_VERSION_4 and external_port)" 

  -> This would allow traffic to go egress into the provider network

  https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1122
  if-return statement was commented out.

  -> I need that connmark/mark restore rule to have ingress replies
  match the existing connections.

  Validate:

  ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t mangle -L neutron-l3-agent-POSTROUTING -nv
  Chain neutron-l3-agent-POSTROUTING (1 references)
  pkts bytes target prot opt in out source destination
  508 27174 CONNMARK all -- * qg-b075d908-66 0.0.0.0/0 0.0.0.0/0 connmark match 0x0/0xffff0000 CONNMARK save mask 0xffff0000

  
  1 entry

  ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t filter -L neutron-l3-agent-scope -nv
  Chain neutron-l3-agent-scope (1 references)
  pkts bytes target prot opt in out source destination
  6588 350K DROP all -- * sg-c46c9df8-06 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4010000/0xffff0000

  1 entry (instead of two before)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1839004/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next