yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #11839
[Bug 1269407] Re: Instance Termination delays in updating port list
Hi Yair,
I looked into this and could reproduce this with your script. The reason
why this occurs is because the events to nova-api are async. When nova
delete is done the instance isn't actually deleted till sometime after
which removes the port. This is why you are getting the error that
multiple ports exist when you go to delete the subnet. In order to do
this you'd need to loop on nova list/show on that instance till it is
gone. Then you can delete the subnet without a problem.
** Changed in: nova
Status: New => Incomplete
** Changed in: neutron
Status: New => Invalid
** Changed in: nova
Status: Incomplete => Invalid
** Changed in: neutron
Assignee: Aaron Rosen (arosen) => (unassigned)
** Changed in: nova
Assignee: Aaron Rosen (arosen) => (unassigned)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1269407
Title:
Instance Termination delays in updating port list
Status in OpenStack Neutron (virtual network service):
Invalid
Status in OpenStack Compute (Nova):
Invalid
Bug description:
When deleting an instance, the port list is not updated immediately.
As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this
subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick
enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
Version:
openstack-nova-api-2014.1-0.5.b1.el6.noarch
openstack-nova-compute-2014.1-0.5.b1.el6.noarch
openstack-nova-scheduler-2014.1-0.5.b1.el6.noarch
openstack-nova-console-2014.1-0.5.b1.el6.noarch
openstack-nova-conductor-2014.1-0.5.b1.el6.noarch
openstack-nova-cert-2014.1-0.5.b1.el6.noarch
python-neutron-2014.1-0.1.b1.el6.noarch
openstack-neutron-2014.1-0.1.b1.el6.noarch
openstack-neutron-openvswitch-2014.1-0.1.b1.el6.noarch
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this
subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1269407/+subscriptions
References
