yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1957794] [NEW] qrouter ns leak while last service port delete because of router gw port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Krzysztof Tomaszewski <1957794@xxxxxxxxxxxxxxxxxx>
Date: Thu, 13 Jan 2022 12:53:59 -0000
Reply-to: Bug 1957794 <1957794@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

While removing last port from the subnet on compute host with DVR then
L3 agent is cleaning unneeded qrouter-* namespaces.

When you have a different (even other user) VM on the same host that has port
from the subnet that your router has a gateway then deleting of qrouter namespaces is not triggered.

Scenario to reproduce:

Two instances multinode devstack master; no dhcp agent (for simplicity);
devstack default DVR router preconfiguration (public net as a default GW, private net as a subnet);
two nodes:
 - devstack1 - dvr_snat node,
 - devstack2 - dvr node

1) create a VM with private network on devstack2 node as a demo user:

(demo)$ openstack server create --net private --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_private
(demo)$ openstack server show test_private -c id
+-------+--------------------------------------+
| Field | Value                                |
+-------+--------------------------------------+
| id    | 7e5bebfd-636d-4416-b2ce-7f16a7b720ca |
+-------+--------------------------------------+
(demo)$ openstack port list --device-id 7e5bebfd-636d-4416-b2ce-7f16a7b720ca -c id
+--------------------------------------+
| ID                                   |
+--------------------------------------+
| d359efe3-8075-483a-90ee-807595d8786a |
+--------------------------------------+

There is proper tap interface and L3 agent creates qrouter-* namespace:

stack@devstack2:~/$ sudo ip netns | grep qr
qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
stack@devstack2:~/$ ip a | grep d359
28: tapd359efe3-80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
stack@devstack2:~$ sudo ovs-vsctl get port tapd359efe3-80 tag
4
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name           tag
-------------- ---
qr-c3ae7e60-aa 4
qr-7f7c0893-f7 4
tapd359efe3-80 4

2) create a VM with public network on devstack2 node as an admin user:

(admin)$ openstack server create --net public --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_public
(admin)$ openstack server show test_public -c OS-EXT-SRV-ATTR:host -c id -c OS-EXT-STS:power_state -c OS-EXT-STS:vm_state
+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| OS-EXT-SRV-ATTR:host   | devstack2                            |
| OS-EXT-STS:power_state | Running                              |
| OS-EXT-STS:vm_state    | active                               |
| id                     | 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 |
+------------------------+--------------------------------------+
(admin)$ openstack port list --device-id 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 -c id
+--------------------------------------+
| ID                                   |
+--------------------------------------+
| dc822c75-715e-4788-9589-3fff05ccc307 |
+--------------------------------------+

stack@devstack2:~$ ip a | grep dc8
14: tapdc822c75-71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000

3) delete demo user test_private VM

(demo)$ openstack server delete test_private

VM is deleted but qrouter-* namespaces stays.

One VM only exist (admin's one):
stack@devstack2:~$ sudo virsh list --all
 Id   Name                State
-----------------------------------
 2    instance-00000007   running

stack@devstack2:~$ sudo ip netns | grep qr
qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
stack@devstack2:~$
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name           tag
-------------- ---
qr-c3ae7e60-aa 4
qr-7f7c0893-f7 4

To clear this namespace you need to full resync L3 agent by agent
restart or by disabling/enabling agent:

(admin)$ openstack network agent list --host devstack2 --agent-type l3 -c ID -c Host
+--------------------------------------+-----------+
| ID                                   | Host      |
+--------------------------------------+-----------+
| 77b01aa0-de3b-4b6b-a40a-08031460a97f | devstack2 |
+--------------------------------------+-----------+

(admin)$ openstack network agent set --disable 77b01aa0-de3b-4b6b-a40a-08031460a97f
(admin)$ openstack network agent set --enable 77b01aa0-de3b-4b6b-a40a-08031460a97f

and qrouter-* namespace disappear:

stack@devstack2:~$ sudo ip netns | grep qr
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name tag
---- ---
stack@devstack2:~$

** Affects: neutron
     Importance: Undecided
     Assignee: Krzysztof Tomaszewski (labedz)
         Status: New


** Tags: l3-dvr-backlog

** Changed in: neutron
     Assignee: (unassigned) => Krzysztof Tomaszewski (labedz)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1957794

Title:
  qrouter ns leak while last service port delete because of router gw
  port

Status in neutron:
  New

Bug description:
  While removing last port from the subnet on compute host with DVR then
  L3 agent is cleaning unneeded qrouter-* namespaces.

  When you have a different (even other user) VM on the same host that has port
  from the subnet that your router has a gateway then deleting of qrouter namespaces is not triggered.

  Scenario to reproduce:

  Two instances multinode devstack master; no dhcp agent (for simplicity);
  devstack default DVR router preconfiguration (public net as a default GW, private net as a subnet);
  two nodes:
   - devstack1 - dvr_snat node,
   - devstack2 - dvr node

  1) create a VM with private network on devstack2 node as a demo user:

  (demo)$ openstack server create --net private --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_private
  (demo)$ openstack server show test_private -c id
  +-------+--------------------------------------+
  | Field | Value                                |
  +-------+--------------------------------------+
  | id    | 7e5bebfd-636d-4416-b2ce-7f16a7b720ca |
  +-------+--------------------------------------+
  (demo)$ openstack port list --device-id 7e5bebfd-636d-4416-b2ce-7f16a7b720ca -c id
  +--------------------------------------+
  | ID                                   |
  +--------------------------------------+
  | d359efe3-8075-483a-90ee-807595d8786a |
  +--------------------------------------+

  There is proper tap interface and L3 agent creates qrouter-*
  namespace:

  stack@devstack2:~/$ sudo ip netns | grep qr
  qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
  stack@devstack2:~/$ ip a | grep d359
  28: tapd359efe3-80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
  stack@devstack2:~$ sudo ovs-vsctl get port tapd359efe3-80 tag
  4
  stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
  name           tag
  -------------- ---
  qr-c3ae7e60-aa 4
  qr-7f7c0893-f7 4
  tapd359efe3-80 4

  2) create a VM with public network on devstack2 node as an admin user:

  (admin)$ openstack server create --net public --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_public
  (admin)$ openstack server show test_public -c OS-EXT-SRV-ATTR:host -c id -c OS-EXT-STS:power_state -c OS-EXT-STS:vm_state
  +------------------------+--------------------------------------+
  | Field                  | Value                                |
  +------------------------+--------------------------------------+
  | OS-EXT-SRV-ATTR:host   | devstack2                            |
  | OS-EXT-STS:power_state | Running                              |
  | OS-EXT-STS:vm_state    | active                               |
  | id                     | 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 |
  +------------------------+--------------------------------------+
  (admin)$ openstack port list --device-id 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 -c id
  +--------------------------------------+
  | ID                                   |
  +--------------------------------------+
  | dc822c75-715e-4788-9589-3fff05ccc307 |
  +--------------------------------------+

  stack@devstack2:~$ ip a | grep dc8
  14: tapdc822c75-71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000

  3) delete demo user test_private VM

  (demo)$ openstack server delete test_private

  VM is deleted but qrouter-* namespaces stays.

  One VM only exist (admin's one):
  stack@devstack2:~$ sudo virsh list --all
   Id   Name                State
  -----------------------------------
   2    instance-00000007   running

  stack@devstack2:~$ sudo ip netns | grep qr
  qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
  stack@devstack2:~$
  stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
  name           tag
  -------------- ---
  qr-c3ae7e60-aa 4
  qr-7f7c0893-f7 4

  To clear this namespace you need to full resync L3 agent by agent
  restart or by disabling/enabling agent:

  (admin)$ openstack network agent list --host devstack2 --agent-type l3 -c ID -c Host
  +--------------------------------------+-----------+
  | ID                                   | Host      |
  +--------------------------------------+-----------+
  | 77b01aa0-de3b-4b6b-a40a-08031460a97f | devstack2 |
  +--------------------------------------+-----------+

  (admin)$ openstack network agent set --disable 77b01aa0-de3b-4b6b-a40a-08031460a97f
  (admin)$ openstack network agent set --enable 77b01aa0-de3b-4b6b-a40a-08031460a97f

  and qrouter-* namespace disappear:

  stack@devstack2:~$ sudo ip netns | grep qr
  stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
  name tag
  ---- ---
  stack@devstack2:~$

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1957794/+subscriptions
Follow ups

[Bug 1957794] Re: qrouter ns leak while last service port delete because of router gw port
From: Krzysztof Tomaszewski, 2022-08-05