yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #08262
[Bug 1269407] [NEW] Instance Termination delays in updating port list
Public bug reported:
When deleting an instance, the port list is not updated immediately. As
a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick
enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x
** Affects: neutron
Importance: Undecided
Status: New
** Affects: nova
Importance: Undecided
Status: New
** Affects: tempest
Importance: Undecided
Status: New
** Attachment added: "bash script to reproduce bug via CLI"
https://bugs.launchpad.net/bugs/1269407/+attachment/3949453/+files/devstack_script.sh
** Also affects: neutron
Importance: Undecided
Status: New
** Also affects: tempest
Importance: Undecided
Status: New
** Description changed:
When deleting an instance, the port list is not updated immediately. As
a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick
enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
+ (*****) might also affect Tempest during tearDown execution
+
How to reproduce:
+ **** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}
-
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
-
- 1 + EXT_NET_NAME=public
- 2 + NET_NAME=my_net
- 3 + SUBNET_NAME=my_subnet
- 4 + ROUTER_NAME=my_router
- 5 + SERVER_NAME=my_server
- 6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
- 7 + MASK=54.0.0
- 8 + SERVER_IP=54.0.0.6
- 9 ++ neutron net-list
- 10 ++ grep public
- 11 ++ awk '{print $2;}'
- 12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
- 13 ++ nova image-list
- 14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
- 15 ++ awk '{print $2;}'
- 16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
- 17 ++ grep -w id
- 18 ++ neutron net-create my_net
- 19 ++ awk '{print $4;}'
- 20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
- 21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
- 22 ++ grep -w id
- 23 ++ awk '{print $4;}'
- 24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
- 25 ++ neutron router-create my_router
- 26 ++ awk '{print $4;}'
- 27 ++ grep -w id
- 28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
- 29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
- 30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
- 31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
- 32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
- 33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
- 34 +--------------------------------------+--------------------------------------+
- 35 | Property | Value |
- 36 +--------------------------------------+--------------------------------------+
- 37 | OS-DCF:diskConfig | MANUAL |
- 38 | OS-EXT-AZ:availability_zone | nova |
- 39 | OS-EXT-STS:power_state | 0 |
- 40 | OS-EXT-STS:task_state | scheduling |
- 41 | OS-EXT-STS:vm_state | building |
- 42 | OS-SRV-USG:launched_at | None |
- 43 | OS-SRV-USG:terminated_at | None |
- 44 | accessIPv4 | |
- 45 | accessIPv6 | |
- 46 | adminPass | 9VJ9XMrtFghW |
- 47 | config_drive | |
- 48 | created | 2014-01-15T12:56:30Z |
- 49 | flavor | m1.small |
- 50 | hostId | |
- 51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
- 52 | image | cirros-0.3.1-x86_64-uec |
- 53 | key_name | None |
- 54 | metadata | {} |
- 55 | name | my_server |
- 56 | os-extended-volumes:volumes_attached | [] |
- 57 | progress | 0 |
- 58 | security_groups | [{"name": "default"}] |
- 59 | status | BUILD |
- 60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
- 61 | updated | 2014-01-15T12:56:30Z |
- 62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
- 63 +--------------------------------------+--------------------------------------+
- 64 ++ nova list
- 65 ++ grep my_server
- 66 ++ awk '{print $2;}'
- 67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
- 68 ++ neutron port-list
- 69 ++ grep 54.0.0.6
- 70 ++ awk '{print $2;}'
- 71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
- 72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
- 73 ++ grep floating_ip_address
- 74 ++ awk '{print $4;}'
- 75 + FLOATING_IP=172.24.4.4
- 76 ++ neutron floatingip-list
- 77 ++ awk '{print $2;}'
- 78 ++ grep 172.24.4.4
- 79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
- 80 + neutron port-list
- 81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
- 82 | id | name | mac_address | fixed_ips |
- 83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
- 84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
- 85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
- 86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
- 87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
- 88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
- 89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
- 90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
- 91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
- 92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
- 93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
- 94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
- 95 + neutron port-list
- 96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
- 97 | id | name | mac_address | fixed_ips |
- 98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
- 99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
+ 1 + EXT_NET_NAME=public
+ 2 + NET_NAME=my_net
+ 3 + SUBNET_NAME=my_subnet
+ 4 + ROUTER_NAME=my_router
+ 5 + SERVER_NAME=my_server
+ 6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
+ 7 + MASK=54.0.0
+ 8 + SERVER_IP=54.0.0.6
+ 9 ++ neutron net-list
+ 10 ++ grep public
+ 11 ++ awk '{print $2;}'
+ 12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
+ 13 ++ nova image-list
+ 14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
+ 15 ++ awk '{print $2;}'
+ 16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
+ 17 ++ grep -w id
+ 18 ++ neutron net-create my_net
+ 19 ++ awk '{print $4;}'
+ 20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
+ 21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
+ 22 ++ grep -w id
+ 23 ++ awk '{print $4;}'
+ 24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
+ 25 ++ neutron router-create my_router
+ 26 ++ awk '{print $4;}'
+ 27 ++ grep -w id
+ 28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
+ 29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
+ 30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
+ 31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
+ 32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
+ 33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
+ 34 +--------------------------------------+--------------------------------------+
+ 35 | Property | Value |
+ 36 +--------------------------------------+--------------------------------------+
+ 37 | OS-DCF:diskConfig | MANUAL |
+ 38 | OS-EXT-AZ:availability_zone | nova |
+ 39 | OS-EXT-STS:power_state | 0 |
+ 40 | OS-EXT-STS:task_state | scheduling |
+ 41 | OS-EXT-STS:vm_state | building |
+ 42 | OS-SRV-USG:launched_at | None |
+ 43 | OS-SRV-USG:terminated_at | None |
+ 44 | accessIPv4 | |
+ 45 | accessIPv6 | |
+ 46 | adminPass | 9VJ9XMrtFghW |
+ 47 | config_drive | |
+ 48 | created | 2014-01-15T12:56:30Z |
+ 49 | flavor | m1.small |
+ 50 | hostId | |
+ 51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
+ 52 | image | cirros-0.3.1-x86_64-uec |
+ 53 | key_name | None |
+ 54 | metadata | {} |
+ 55 | name | my_server |
+ 56 | os-extended-volumes:volumes_attached | [] |
+ 57 | progress | 0 |
+ 58 | security_groups | [{"name": "default"}] |
+ 59 | status | BUILD |
+ 60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
+ 61 | updated | 2014-01-15T12:56:30Z |
+ 62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
+ 63 +--------------------------------------+--------------------------------------+
+ 64 ++ nova list
+ 65 ++ grep my_server
+ 66 ++ awk '{print $2;}'
+ 67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
+ 68 ++ neutron port-list
+ 69 ++ grep 54.0.0.6
+ 70 ++ awk '{print $2;}'
+ 71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
+ 72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
+ 73 ++ grep floating_ip_address
+ 74 ++ awk '{print $4;}'
+ 75 + FLOATING_IP=172.24.4.4
+ 76 ++ neutron floatingip-list
+ 77 ++ awk '{print $2;}'
+ 78 ++ grep 172.24.4.4
+ 79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+ 80 + neutron port-list
+ 81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+ 82 | id | name | mac_address | fixed_ips |
+ 83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+ 84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
+ 85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
+ 86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
+ 87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
+ 88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
+ 89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+ 90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+ 91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+ 92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
+ 93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
+ 94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
+ 95 + neutron port-list
+ 96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+ 97 | id | name | mac_address | fixed_ips |
+ 98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+ 99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1269407
Title:
Instance Termination delays in updating port list
Status in OpenStack Neutron (virtual network service):
New
Status in OpenStack Compute (Nova):
New
Status in Tempest:
New
Bug description:
When deleting an instance, the port list is not updated immediately.
As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this
subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick
enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this
subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1269407/+subscriptions
Follow ups
References
