yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #77360
[Bug 1818805] [NEW] Conntrack rules in the qrouter are not deleted when a fip is removed with dvr
Public bug reported:
If a fip ip is removed of a network with a distributed router:
openstack server remove floating ip X
The conntrack rules aren't deleted in the qrouter and the qrouter continues doing nating of the ongoing connections.
overcloud) [stack@undercloud-0 ~]$ openstack router show router
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2019-02-20T15:46:53Z |
| description | |
| distributed | True |
| external_gateway_info | {"network_id": "15a5c01e-4e42-4890-a850-db4f97bb5834", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c59ae813-1df7-4a14-9eba-be2e35afa13e", "ip_address": "10.0.0.214"}]} |
| flavor_id | None |
| ha | False |
| id | d01c89b0-c2df-46e2-9c12-8d14b1c8ce9a |
| interfaces_info | [{"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.8", "port_id": "06c6e9d3-2c6b-40b8-8919-92be6efd0153"}, {"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.1", "port_id": "c47b0417-7dbe-4434-8c50-72a78e6335a1"}] |
| name | router |
| project_id | 9447276fedbf4c4eab15494f8d187d97 |
| revision_number | 13 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2019-03-05T11:31:34Z |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
| 8aad2992-b068-4378-83f8-965b59d04d8d | selfservice2-instance4 | ACTIVE | selfservice2=10.2.0.12, 10.0.0.210 | cirros | cirros |
| 028e0696-e666-4c09-802a-49a126a6346d | selfservice2-instance3 | ACTIVE | selfservice2=10.2.0.27 | cirros | cirros |
| 46432868-4d11-4c9d-a910-ddf246c78378 | selfservice2-instance2 | ACTIVE | selfservice2=10.2.0.14 | cirros | cirros |
| fca6541e-d846-45fd-8970-8ba27ff708d6 | selfservice2-instance1 | ACTIVE | selfservice2=10.2.0.30 | cirros | cirros |
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:6e:0e:f7 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.12/24 brd 10.2.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe6e:ef7/64 scope link
valid_lft forever preferred_lft forever
$ hostname
selfservice2-instance4
$
$
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=49.554 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=48.893 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=48.897 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=48.988 ms
....
[heat-admin@compute-1 ~]$ sudo su
[root@compute-1 heat-admin]# yum install conntrack-tools
....
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
(overcloud) [stack@undercloud-0 ~]$ openstack server remove floating ip selfservice2-instance4 10.0.0.210
(overcloud) [stack@undercloud-0 ~]$ date
Wed Mar 6 04:59:40 EST 2019
(overcloud) [stack@undercloud-0 ~]$
Then connectivity is lost:
64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms
64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms
because conntrack flow is not deleted:
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]# date
Wed Mar 6 10:02:04 UTC 2019
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
If I delete the flow the connectiviy is recovered(same behavior that
without dvr):
[root@compute-1 heat-admin]# conntrack -D
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted.
[root@compute-1 heat-admin]# conntrack -L
icmp 1 28 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 [UNREPLIED] src=8.8.8.8 dst=10.2.0.12 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
....
64 bytes from 8.8.8.8: seq=812 ttl=116 time=48.791 ms
64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms
64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms
64 bytes from 8.8.8.8: seq=980 ttl=117 time=49.979 ms
64 bytes from 8.8.8.8: seq=981 ttl=117 time=49.164 ms
64 bytes from 8.8.8.8: seq=982 ttl=117 time=49.524 ms
64 bytes from 8.8.8.8: seq=983 ttl=117 time=49.143 ms
....
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1818805
Title:
Conntrack rules in the qrouter are not deleted when a fip is removed
with dvr
Status in neutron:
New
Bug description:
If a fip ip is removed of a network with a distributed router:
openstack server remove floating ip X
The conntrack rules aren't deleted in the qrouter and the qrouter continues doing nating of the ongoing connections.
overcloud) [stack@undercloud-0 ~]$ openstack router show router
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2019-02-20T15:46:53Z |
| description | |
| distributed | True |
| external_gateway_info | {"network_id": "15a5c01e-4e42-4890-a850-db4f97bb5834", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c59ae813-1df7-4a14-9eba-be2e35afa13e", "ip_address": "10.0.0.214"}]} |
| flavor_id | None |
| ha | False |
| id | d01c89b0-c2df-46e2-9c12-8d14b1c8ce9a |
| interfaces_info | [{"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.8", "port_id": "06c6e9d3-2c6b-40b8-8919-92be6efd0153"}, {"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.1", "port_id": "c47b0417-7dbe-4434-8c50-72a78e6335a1"}] |
| name | router |
| project_id | 9447276fedbf4c4eab15494f8d187d97 |
| revision_number | 13 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2019-03-05T11:31:34Z |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
| 8aad2992-b068-4378-83f8-965b59d04d8d | selfservice2-instance4 | ACTIVE | selfservice2=10.2.0.12, 10.0.0.210 | cirros | cirros |
| 028e0696-e666-4c09-802a-49a126a6346d | selfservice2-instance3 | ACTIVE | selfservice2=10.2.0.27 | cirros | cirros |
| 46432868-4d11-4c9d-a910-ddf246c78378 | selfservice2-instance2 | ACTIVE | selfservice2=10.2.0.14 | cirros | cirros |
| fca6541e-d846-45fd-8970-8ba27ff708d6 | selfservice2-instance1 | ACTIVE | selfservice2=10.2.0.30 | cirros | cirros |
+--------------------------------------+------------------------+--------+------------------------------------+--------+--------+
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:6e:0e:f7 brd ff:ff:ff:ff:ff:ff
inet 10.2.0.12/24 brd 10.2.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe6e:ef7/64 scope link
valid_lft forever preferred_lft forever
$ hostname
selfservice2-instance4
$
$
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=49.554 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=48.893 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=48.897 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=48.988 ms
....
[heat-admin@compute-1 ~]$ sudo su
[root@compute-1 heat-admin]# yum install conntrack-tools
....
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
(overcloud) [stack@undercloud-0 ~]$ openstack server remove floating ip selfservice2-instance4 10.0.0.210
(overcloud) [stack@undercloud-0 ~]$ date
Wed Mar 6 04:59:40 EST 2019
(overcloud) [stack@undercloud-0 ~]$
Then connectivity is lost:
64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms
64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms
because conntrack flow is not deleted:
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]# date
Wed Mar 6 10:02:04 UTC 2019
[root@compute-1 heat-admin]# conntrack -L
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
If I delete the flow the connectiviy is recovered(same behavior that
without dvr):
[root@compute-1 heat-admin]# conntrack -D
icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted.
[root@compute-1 heat-admin]# conntrack -L
icmp 1 28 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 [UNREPLIED] src=8.8.8.8 dst=10.2.0.12 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
[root@compute-1 heat-admin]#
....
64 bytes from 8.8.8.8: seq=812 ttl=116 time=48.791 ms
64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms
64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms
64 bytes from 8.8.8.8: seq=980 ttl=117 time=49.979 ms
64 bytes from 8.8.8.8: seq=981 ttl=117 time=49.164 ms
64 bytes from 8.8.8.8: seq=982 ttl=117 time=49.524 ms
64 bytes from 8.8.8.8: seq=983 ttl=117 time=49.143 ms
....
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1818805/+subscriptions
Follow ups
