yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #23351
[Bug 1377156] [NEW] fg- device is not deleted after the deletion of the last VM on the compute node
Public bug reported:
The external gateway port in the fip- namespace on a compute node is not
removed after the user deleted the last VM running on the node.
How to reproduce the problem:
1. SETUP:
* Use devstack to start up the controller node. In local.conf, Q_DVR_MODE=dvr_snat.
* Use devstack to setup a compute node. In local.conf, Q_DVR_MODE=dvr.
At the start, there are no VMs hosted on the compute node. The fip
namespace hasn't been created yet.
2. Create a network and subnet.
3. Create a router, add the subnet to the router using the "neutron router-interface-add" command
4. Tie the router to the external network using the "neutron router-gateway-set" command.
5. Boot up a VM using the network, assign a floatingip to the VM. The VM should be pingable using the floating IP. Note the fg- device in the fip namespace on the compute node. There is a corresponding port in the database.
6. Now delete the VM.
7. The VM is deleted. Neutron port-list shows the fg- port is also deleted. However, the fg- device still remains in the fip namespace on the compute node and the fip namespace isn't removed.
For example:
1. Created a VM and assigned a floating IP to the VM, The VM is pingable. The compute node is running only user-1-vm1:
stack@DVR-Controller:~/DEVSTACK/user-1$ nova list
+--------------------------------------+------------+--------+------------+-------------+--------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+--------------------+
| ff195387-5bff-4c4d-a639-3891a049c811 | user-1-vm1 | ACTIVE | - | Running | user-1net=10.1.2.3 |
+--------------------------------------+------------+--------+------------+-------------+--------------------+
stack@DVR-Controller:~/DEVSTACK/user-1$ neutron floatingip-create --port-id a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| fixed_ip_address | 10.1.2.3 |
| floating_ip_address | 10.127.10.226 |
| floating_network_id | 59c7a096-b07c-4d09-be90-31867aed00c6 |
| id | 0173a17f-0413-45ef-9ea5-0ecd84aa107a |
| port_id | a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 |
| router_id | 18296cb8-f6b1-4d24-a42a-b9ede7aa207c |
| status | DOWN |
| tenant_id | f447c5209cbe4e69a243a58b1310b32d |
+---------------------+--------------------------------------+
stack@DVR-Controller:~/DEVSTACK/user-1$ neutron port-list^C
stack@DVR-Controller:~/DEVSTACK/user-1$ ping 10.127.10.226
PING 10.127.10.226 (10.127.10.226) 56(84) bytes of data.
64 bytes from 10.127.10.226: icmp_seq=1 ttl=62 time=159 ms
64 bytes from 10.127.10.226: icmp_seq=2 ttl=62 time=1.57 ms
^C
--- 10.127.10.226 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.578/80.741/159.905/79.164 ms
2. The external gateway port on the compute node DVR-CN2 is 5cb96efd-eb83-43b0-9fbc-c0003edb1302:
stack@DVR-Controller:~/DEVSTACK/manage$ ./os_admin neutron port-list
/home/stack/DEVSTACK/manage
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0b51a8fc-6737-4b6f-b640-4338e6fe4652 | | fa:16:3e:e2:3f:f7 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.2"} |
| 1b649bc2-7d3c-4e8f-b3fb-80ca12bbe65b | | fa:16:3e:c3:6c:3d | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.226"} |
| 24426b5f-cafa-4688-b5e5-0be38c14408f | | fa:16:3e:c2:c4:83 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.1"} |
| 3813c08c-8c40-46d3-a99a-4bba9a91e2ba | | fa:16:3e:d1:27:e0 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.225"} |
| 3b4119fd-4b69-4310-817e-9b0b8d499d06 | | fa:16:3e:ed:a7:65 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.224"} |
| 5cb96efd-eb83-43b0-9fbc-c0003edb1302 | | fa:16:3e:57:e7:76 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.227"} |
| 686b5139-dc0b-46d8-bc05-b474f973194f | | fa:16:3e:31:bc:3b | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.4"} |
| a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 | | fa:16:3e:64:76:26 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.3"} |
| c1dc81fb-3331-440c-92b2-26f051454689 | | fa:16:3e:4b:29:b4 | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.2"} |
| c3cd18c2-280a-45f9-b3b1-efdfed10d7e1 | | fa:16:3e:df:1d:da | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
3. Delete the VM, The VM is deleted and the floating IP is disassociated.
stack@DVR-Controller:~/DEVSTACK/manage$ nova delete user-1-vm1
Request to delete server user-1-vm1 has been accepted.
stack@DVR-Controller:~/DEVSTACK/manage$ neutron floatingip-list
+--------------------------------------+------------------+---------------------+---------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+---------+
| 0173a17f-0413-45ef-9ea5-0ecd84aa107a | | 10.127.10.226 | |
+--------------------------------------+------------------+---------------------+---------+
stack@DVR-Controller:~/DEVSTACK/manage$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
4. HOWEVER, on the compute that hosted the only VM user-1-vm2, the fip namespace shows that device for port 5cb96efd-eb83-43b0-9fbc-c0003edb1302 still exists even
though it has already been deleted from the database when the VM is
deleted.
The fip- namespace on the compute node after the VM is deleted.
stack@DVR-CN2:~/DEVSTACK/manage$ sudo ip netns exec fip-59c7a096-b07c-4d09-be90-31867aed00c6 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
110: fg-5cb96efd-eb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:57:e7:76 brd ff:ff:ff:ff:ff:ff
inet 10.127.10.227/24 brd 10.127.10.255 scope global fg-5cb96efd-eb
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe57:e776/64 scope link
valid_lft forever preferred_lft forever
The list of ports after the VM deletion:
stack@DVR-Controller:~/DEVSTACK/manage$ ./os_admin neutron port-list
/home/stack/DEVSTACK/manage
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0b51a8fc-6737-4b6f-b640-4338e6fe4652 | | fa:16:3e:e2:3f:f7 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.2"} |
| 1b649bc2-7d3c-4e8f-b3fb-80ca12bbe65b | | fa:16:3e:c3:6c:3d | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.226"} |
| 24426b5f-cafa-4688-b5e5-0be38c14408f | | fa:16:3e:c2:c4:83 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.1"} |
| 3813c08c-8c40-46d3-a99a-4bba9a91e2ba | | fa:16:3e:d1:27:e0 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.225"} |
| 3b4119fd-4b69-4310-817e-9b0b8d499d06 | | fa:16:3e:ed:a7:65 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.224"} |
| 686b5139-dc0b-46d8-bc05-b474f973194f | | fa:16:3e:31:bc:3b | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.4"} |
| c1dc81fb-3331-440c-92b2-26f051454689 | | fa:16:3e:4b:29:b4 | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.2"} |
| c3cd18c2-280a-45f9-b3b1-efdfed10d7e1 | | fa:16:3e:df:1d:da | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
** Affects: neutron
Importance: Undecided
Assignee: Stephen Ma (stephen-ma)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Stephen Ma (stephen-ma)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1377156
Title:
fg- device is not deleted after the deletion of the last VM on the
compute node
Status in OpenStack Neutron (virtual network service):
New
Bug description:
The external gateway port in the fip- namespace on a compute node is
not removed after the user deleted the last VM running on the node.
How to reproduce the problem:
1. SETUP:
* Use devstack to start up the controller node. In local.conf, Q_DVR_MODE=dvr_snat.
* Use devstack to setup a compute node. In local.conf, Q_DVR_MODE=dvr.
At the start, there are no VMs hosted on the compute node. The fip
namespace hasn't been created yet.
2. Create a network and subnet.
3. Create a router, add the subnet to the router using the "neutron router-interface-add" command
4. Tie the router to the external network using the "neutron router-gateway-set" command.
5. Boot up a VM using the network, assign a floatingip to the VM. The VM should be pingable using the floating IP. Note the fg- device in the fip namespace on the compute node. There is a corresponding port in the database.
6. Now delete the VM.
7. The VM is deleted. Neutron port-list shows the fg- port is also deleted. However, the fg- device still remains in the fip namespace on the compute node and the fip namespace isn't removed.
For example:
1. Created a VM and assigned a floating IP to the VM, The VM is pingable. The compute node is running only user-1-vm1:
stack@DVR-Controller:~/DEVSTACK/user-1$ nova list
+--------------------------------------+------------+--------+------------+-------------+--------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+--------------------+
| ff195387-5bff-4c4d-a639-3891a049c811 | user-1-vm1 | ACTIVE | - | Running | user-1net=10.1.2.3 |
+--------------------------------------+------------+--------+------------+-------------+--------------------+
stack@DVR-Controller:~/DEVSTACK/user-1$ neutron floatingip-create --port-id a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| fixed_ip_address | 10.1.2.3 |
| floating_ip_address | 10.127.10.226 |
| floating_network_id | 59c7a096-b07c-4d09-be90-31867aed00c6 |
| id | 0173a17f-0413-45ef-9ea5-0ecd84aa107a |
| port_id | a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 |
| router_id | 18296cb8-f6b1-4d24-a42a-b9ede7aa207c |
| status | DOWN |
| tenant_id | f447c5209cbe4e69a243a58b1310b32d |
+---------------------+--------------------------------------+
stack@DVR-Controller:~/DEVSTACK/user-1$ neutron port-list^C
stack@DVR-Controller:~/DEVSTACK/user-1$ ping 10.127.10.226
PING 10.127.10.226 (10.127.10.226) 56(84) bytes of data.
64 bytes from 10.127.10.226: icmp_seq=1 ttl=62 time=159 ms
64 bytes from 10.127.10.226: icmp_seq=2 ttl=62 time=1.57 ms
^C
--- 10.127.10.226 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.578/80.741/159.905/79.164 ms
2. The external gateway port on the compute node DVR-CN2 is 5cb96efd-eb83-43b0-9fbc-c0003edb1302:
stack@DVR-Controller:~/DEVSTACK/manage$ ./os_admin neutron port-list
/home/stack/DEVSTACK/manage
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0b51a8fc-6737-4b6f-b640-4338e6fe4652 | | fa:16:3e:e2:3f:f7 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.2"} |
| 1b649bc2-7d3c-4e8f-b3fb-80ca12bbe65b | | fa:16:3e:c3:6c:3d | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.226"} |
| 24426b5f-cafa-4688-b5e5-0be38c14408f | | fa:16:3e:c2:c4:83 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.1"} |
| 3813c08c-8c40-46d3-a99a-4bba9a91e2ba | | fa:16:3e:d1:27:e0 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.225"} |
| 3b4119fd-4b69-4310-817e-9b0b8d499d06 | | fa:16:3e:ed:a7:65 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.224"} |
| 5cb96efd-eb83-43b0-9fbc-c0003edb1302 | | fa:16:3e:57:e7:76 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.227"} |
| 686b5139-dc0b-46d8-bc05-b474f973194f | | fa:16:3e:31:bc:3b | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.4"} |
| a22d3f90-2af8-4c7e-8bfa-c37bc3a73f01 | | fa:16:3e:64:76:26 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.3"} |
| c1dc81fb-3331-440c-92b2-26f051454689 | | fa:16:3e:4b:29:b4 | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.2"} |
| c3cd18c2-280a-45f9-b3b1-efdfed10d7e1 | | fa:16:3e:df:1d:da | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
3. Delete the VM, The VM is deleted and the floating IP is disassociated.
stack@DVR-Controller:~/DEVSTACK/manage$ nova delete user-1-vm1
Request to delete server user-1-vm1 has been accepted.
stack@DVR-Controller:~/DEVSTACK/manage$ neutron floatingip-list
+--------------------------------------+------------------+---------------------+---------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+---------+
| 0173a17f-0413-45ef-9ea5-0ecd84aa107a | | 10.127.10.226 | |
+--------------------------------------+------------------+---------------------+---------+
stack@DVR-Controller:~/DEVSTACK/manage$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
4. HOWEVER, on the compute that hosted the only VM user-1-vm2, the fip namespace shows that device for port 5cb96efd-eb83-43b0-9fbc-c0003edb1302 still exists even
though it has already been deleted from the database when the VM is
deleted.
The fip- namespace on the compute node after the VM is deleted.
stack@DVR-CN2:~/DEVSTACK/manage$ sudo ip netns exec fip-59c7a096-b07c-4d09-be90-31867aed00c6 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
110: fg-5cb96efd-eb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:57:e7:76 brd ff:ff:ff:ff:ff:ff
inet 10.127.10.227/24 brd 10.127.10.255 scope global fg-5cb96efd-eb
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe57:e776/64 scope link
valid_lft forever preferred_lft forever
The list of ports after the VM deletion:
stack@DVR-Controller:~/DEVSTACK/manage$ ./os_admin neutron port-list
/home/stack/DEVSTACK/manage
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 0b51a8fc-6737-4b6f-b640-4338e6fe4652 | | fa:16:3e:e2:3f:f7 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.2"} |
| 1b649bc2-7d3c-4e8f-b3fb-80ca12bbe65b | | fa:16:3e:c3:6c:3d | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.226"} |
| 24426b5f-cafa-4688-b5e5-0be38c14408f | | fa:16:3e:c2:c4:83 | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.1"} |
| 3813c08c-8c40-46d3-a99a-4bba9a91e2ba | | fa:16:3e:d1:27:e0 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.225"} |
| 3b4119fd-4b69-4310-817e-9b0b8d499d06 | | fa:16:3e:ed:a7:65 | {"subnet_id": "6502fec5-ae38-4b7e-a84d-d0779e85ecf7", "ip_address": "10.127.10.224"} |
| 686b5139-dc0b-46d8-bc05-b474f973194f | | fa:16:3e:31:bc:3b | {"subnet_id": "3cd21767-38b5-4cef-9a17-60a03e1400ef", "ip_address": "10.1.2.4"} |
| c1dc81fb-3331-440c-92b2-26f051454689 | | fa:16:3e:4b:29:b4 | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.2"} |
| c3cd18c2-280a-45f9-b3b1-efdfed10d7e1 | | fa:16:3e:df:1d:da | {"subnet_id": "e7412ed4-e037-42d7-bbca-d2cefef68bec", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1377156/+subscriptions
Follow ups
References
