← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1269407] [NEW] Instance Termination delays in updating port list

 

Public bug reported:

When deleting an instance, the port list is not updated immediately. As
a result - deleting net/subnet fails with error:

409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}

(*) Happens only on automated scripts, since manual input isn't quick
enough to catch this.

(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.

(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure

(*****) might also affect Tempest during tearDown execution

How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists

Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM

TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net

Expected Result:
subnet/net should be successfully deleted.

Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"

409-{u'NeutronError': {u'message': u'Unable to complete operation on
subnet UUID. One or more ports have an IP allocation from this subnet.',
u'type': u'SubnetInUse', u'detail': u''}}

script log:

line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update

  1 + EXT_NET_NAME=public
  2 + NET_NAME=my_net
  3 + SUBNET_NAME=my_subnet
  4 + ROUTER_NAME=my_router
  5 + SERVER_NAME=my_server
  6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
  7 + MASK=54.0.0
  8 + SERVER_IP=54.0.0.6
  9 ++ neutron net-list
 10 ++ grep public
 11 ++ awk '{print $2;}'
 12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
 13 ++ nova image-list
 14 ++ grep -w ' cirros-0.3.1-x86_64-uec  '
 15 ++ awk '{print $2;}'
 16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
 17 ++ grep -w id
 18 ++ neutron net-create my_net
 19 ++ awk '{print $4;}'
 20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
 21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
 22 ++ grep -w id
 23 ++ awk '{print $4;}'
 24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
 25 ++ neutron router-create my_router
 26 ++ awk '{print $4;}'
 27 ++ grep -w id
 28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
 29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
 30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
 31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
 32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
 33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
 34 +--------------------------------------+--------------------------------------+
 35 | Property                             | Value                                |
 36 +--------------------------------------+--------------------------------------+
 37 | OS-DCF:diskConfig                    | MANUAL                               |
 38 | OS-EXT-AZ:availability_zone          | nova                                 |
 39 | OS-EXT-STS:power_state               | 0                                    |
 40 | OS-EXT-STS:task_state                | scheduling                           |
 41 | OS-EXT-STS:vm_state                  | building                             |
 42 | OS-SRV-USG:launched_at               | None                                 |
 43 | OS-SRV-USG:terminated_at             | None                                 |
 44 | accessIPv4                           |                                      |
 45 | accessIPv6                           |                                      |
 46 | adminPass                            | 9VJ9XMrtFghW                         |
 47 | config_drive                         |                                      |
 48 | created                              | 2014-01-15T12:56:30Z                 |
 49 | flavor                               | m1.small                             |
 50 | hostId                               |                                      |
 51 | id                                   | d09bbee1-599c-4042-a409-a90e0ae80e3c |
 52 | image                                | cirros-0.3.1-x86_64-uec              |
 53 | key_name                             | None                                 |
 54 | metadata                             | {}                                   |
 55 | name                                 | my_server                            |
 56 | os-extended-volumes:volumes_attached | []                                   |
 57 | progress                             | 0                                    |
 58 | security_groups                      | [{"name": "default"}]                |
 59 | status                               | BUILD                                |
 60 | tenant_id                            | 66293febf7164c849b694a8d3f14cc1a     |
 61 | updated                              | 2014-01-15T12:56:30Z                 |
 62 | user_id                              | e895d5bf0dc64d44b707b9fbf5cbf6e0     |
 63 +--------------------------------------+--------------------------------------+
 64 ++ nova list
 65 ++ grep my_server
 66 ++ awk '{print $2;}'
 67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
 68 ++ neutron port-list
 69 ++ grep 54.0.0.6
 70 ++ awk '{print $2;}'
 71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
 72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
 73 ++ grep floating_ip_address
 74 ++ awk '{print $4;}'
 75 + FLOATING_IP=172.24.4.4
 76 ++ neutron floatingip-list
 77 ++ awk '{print $2;}'
 78 ++ grep 172.24.4.4
 79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
 80 + neutron port-list
 81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
 82 | id                                   | name | mac_address       | fixed_ips                                                                       |
 83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
 84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
 85 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
 86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
 87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
 88 | cafd4161-f840-4c87-a80b-71b0ef374b9e |      | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
 89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
 90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
 91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
 92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
 93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
 94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
 95 + neutron port-list
 96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
 97 | id                                   | name | mac_address       | fixed_ips                                                                       |
 98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
 99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id                                   | name | mac_address       | fixed_ips                                                                       |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: tempest
     Importance: Undecided
         Status: New

** Attachment added: "bash script to reproduce bug via CLI"
   https://bugs.launchpad.net/bugs/1269407/+attachment/3949453/+files/devstack_script.sh

** Also affects: neutron
   Importance: Undecided
       Status: New

** Also affects: tempest
   Importance: Undecided
       Status: New

** Description changed:

  When deleting an instance, the port list is not updated immediately. As
  a result - deleting net/subnet fails with error:
  
  409-{u'NeutronError': {u'message': u'Unable to complete operation on
  subnet UUID. One or more ports have an IP allocation from this subnet.',
  u'type': u'SubnetInUse', u'detail': u''}}
  
  (*) Happens only on automated scripts, since manual input isn't quick
  enough to catch this.
  
  (**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
  my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
  
  (***) nova delete command returns BEFORE instance is deleted:
  > nova delete $instance_id ; nova show $instance_id
  will return the instance without failure
  
+ (*****) might also affect Tempest during tearDown execution
+ 
  How to reproduce:
+ **** script attached ****
  assumes:
  1. external network exists with floating ip range available ("public")
  2. image exists
  
  Setup:
  1 create network "private"
  2. create subnet
  3. create router:
  3.1 set router gateway to "public"
  3.2 set router interface to "private"
  4. create VM
  5. assign Floating IP to VM
  
  TearDown
  1. Delete / Disassociate Floating IP
  2. Delete VM
  3. detach router interface from subnet (router-interface-delete)
  4. Delete subnet/net
  
  Expected Result:
  subnet/net should be successfully deleted.
  
  Actual Results:
  "Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
  
  409-{u'NeutronError': {u'message': u'Unable to complete operation on
  subnet UUID. One or more ports have an IP allocation from this subnet.',
  u'type': u'SubnetInUse', u'detail': u''}}
  
- 
  script log:
  
  line 101 - VM port still in port list even though VM was deleted
  line 105 - subnet fails to delete
  line 117 - network successfully deleted after enough time passed for port list to update
  
- 
-   1 + EXT_NET_NAME=public
-   2 + NET_NAME=my_net
-   3 + SUBNET_NAME=my_subnet
-   4 + ROUTER_NAME=my_router
-   5 + SERVER_NAME=my_server
-   6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
-   7 + MASK=54.0.0
-   8 + SERVER_IP=54.0.0.6
-   9 ++ neutron net-list
-  10 ++ grep public
-  11 ++ awk '{print $2;}'
-  12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
-  13 ++ nova image-list
-  14 ++ grep -w ' cirros-0.3.1-x86_64-uec  '
-  15 ++ awk '{print $2;}'
-  16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
-  17 ++ grep -w id
-  18 ++ neutron net-create my_net
-  19 ++ awk '{print $4;}'
-  20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
-  21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
-  22 ++ grep -w id
-  23 ++ awk '{print $4;}'
-  24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
-  25 ++ neutron router-create my_router
-  26 ++ awk '{print $4;}'
-  27 ++ grep -w id
-  28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
-  29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
-  30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
-  31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
-  32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
-  33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
-  34 +--------------------------------------+--------------------------------------+
-  35 | Property                             | Value                                |
-  36 +--------------------------------------+--------------------------------------+
-  37 | OS-DCF:diskConfig                    | MANUAL                               |
-  38 | OS-EXT-AZ:availability_zone          | nova                                 |
-  39 | OS-EXT-STS:power_state               | 0                                    |
-  40 | OS-EXT-STS:task_state                | scheduling                           |
-  41 | OS-EXT-STS:vm_state                  | building                             |
-  42 | OS-SRV-USG:launched_at               | None                                 |
-  43 | OS-SRV-USG:terminated_at             | None                                 |
-  44 | accessIPv4                           |                                      |
-  45 | accessIPv6                           |                                      |
-  46 | adminPass                            | 9VJ9XMrtFghW                         |
-  47 | config_drive                         |                                      |
-  48 | created                              | 2014-01-15T12:56:30Z                 |
-  49 | flavor                               | m1.small                             |
-  50 | hostId                               |                                      |
-  51 | id                                   | d09bbee1-599c-4042-a409-a90e0ae80e3c |
-  52 | image                                | cirros-0.3.1-x86_64-uec              |
-  53 | key_name                             | None                                 |
-  54 | metadata                             | {}                                   |
-  55 | name                                 | my_server                            |
-  56 | os-extended-volumes:volumes_attached | []                                   |
-  57 | progress                             | 0                                    |
-  58 | security_groups                      | [{"name": "default"}]                |
-  59 | status                               | BUILD                                |
-  60 | tenant_id                            | 66293febf7164c849b694a8d3f14cc1a     |
-  61 | updated                              | 2014-01-15T12:56:30Z                 |
-  62 | user_id                              | e895d5bf0dc64d44b707b9fbf5cbf6e0     |
-  63 +--------------------------------------+--------------------------------------+
-  64 ++ nova list
-  65 ++ grep my_server
-  66 ++ awk '{print $2;}'
-  67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
-  68 ++ neutron port-list
-  69 ++ grep 54.0.0.6
-  70 ++ awk '{print $2;}'
-  71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
-  72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
-  73 ++ grep floating_ip_address
-  74 ++ awk '{print $4;}'
-  75 + FLOATING_IP=172.24.4.4
-  76 ++ neutron floatingip-list
-  77 ++ awk '{print $2;}'
-  78 ++ grep 172.24.4.4
-  79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
-  80 + neutron port-list
-  81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
-  82 | id                                   | name | mac_address       | fixed_ips                                                                       |
-  83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
-  84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
-  85 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
-  86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
-  87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
-  88 | cafd4161-f840-4c87-a80b-71b0ef374b9e |      | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
-  89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
-  90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
-  91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
-  92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
-  93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
-  94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
-  95 + neutron port-list
-  96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
-  97 | id                                   | name | mac_address       | fixed_ips                                                                       |
-  98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
-  99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
+   1 + EXT_NET_NAME=public
+   2 + NET_NAME=my_net
+   3 + SUBNET_NAME=my_subnet
+   4 + ROUTER_NAME=my_router
+   5 + SERVER_NAME=my_server
+   6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
+   7 + MASK=54.0.0
+   8 + SERVER_IP=54.0.0.6
+   9 ++ neutron net-list
+  10 ++ grep public
+  11 ++ awk '{print $2;}'
+  12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
+  13 ++ nova image-list
+  14 ++ grep -w ' cirros-0.3.1-x86_64-uec  '
+  15 ++ awk '{print $2;}'
+  16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
+  17 ++ grep -w id
+  18 ++ neutron net-create my_net
+  19 ++ awk '{print $4;}'
+  20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
+  21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
+  22 ++ grep -w id
+  23 ++ awk '{print $4;}'
+  24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
+  25 ++ neutron router-create my_router
+  26 ++ awk '{print $4;}'
+  27 ++ grep -w id
+  28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
+  29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
+  30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
+  31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
+  32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
+  33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
+  34 +--------------------------------------+--------------------------------------+
+  35 | Property                             | Value                                |
+  36 +--------------------------------------+--------------------------------------+
+  37 | OS-DCF:diskConfig                    | MANUAL                               |
+  38 | OS-EXT-AZ:availability_zone          | nova                                 |
+  39 | OS-EXT-STS:power_state               | 0                                    |
+  40 | OS-EXT-STS:task_state                | scheduling                           |
+  41 | OS-EXT-STS:vm_state                  | building                             |
+  42 | OS-SRV-USG:launched_at               | None                                 |
+  43 | OS-SRV-USG:terminated_at             | None                                 |
+  44 | accessIPv4                           |                                      |
+  45 | accessIPv6                           |                                      |
+  46 | adminPass                            | 9VJ9XMrtFghW                         |
+  47 | config_drive                         |                                      |
+  48 | created                              | 2014-01-15T12:56:30Z                 |
+  49 | flavor                               | m1.small                             |
+  50 | hostId                               |                                      |
+  51 | id                                   | d09bbee1-599c-4042-a409-a90e0ae80e3c |
+  52 | image                                | cirros-0.3.1-x86_64-uec              |
+  53 | key_name                             | None                                 |
+  54 | metadata                             | {}                                   |
+  55 | name                                 | my_server                            |
+  56 | os-extended-volumes:volumes_attached | []                                   |
+  57 | progress                             | 0                                    |
+  58 | security_groups                      | [{"name": "default"}]                |
+  59 | status                               | BUILD                                |
+  60 | tenant_id                            | 66293febf7164c849b694a8d3f14cc1a     |
+  61 | updated                              | 2014-01-15T12:56:30Z                 |
+  62 | user_id                              | e895d5bf0dc64d44b707b9fbf5cbf6e0     |
+  63 +--------------------------------------+--------------------------------------+
+  64 ++ nova list
+  65 ++ grep my_server
+  66 ++ awk '{print $2;}'
+  67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
+  68 ++ neutron port-list
+  69 ++ grep 54.0.0.6
+  70 ++ awk '{print $2;}'
+  71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
+  72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
+  73 ++ grep floating_ip_address
+  74 ++ awk '{print $4;}'
+  75 + FLOATING_IP=172.24.4.4
+  76 ++ neutron floatingip-list
+  77 ++ awk '{print $2;}'
+  78 ++ grep 172.24.4.4
+  79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+  80 + neutron port-list
+  81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+  82 | id                                   | name | mac_address       | fixed_ips                                                                       |
+  83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+  84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
+  85 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
+  86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
+  87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
+  88 | cafd4161-f840-4c87-a80b-71b0ef374b9e |      | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
+  89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+  90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+  91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
+  92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
+  93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
+  94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
+  95 + neutron port-list
+  96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+  97 | id                                   | name | mac_address       | fixed_ips                                                                       |
+  98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
+  99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
  100 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
  101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
  102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
  103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
  105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
  106 + neutron port-list
  107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  108 | id                                   | name | mac_address       | fixed_ips                                                                       |
  109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
  111 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
  112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
  113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
  115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
  116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
  117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
  118 + set +x

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1269407

Title:
  Instance Termination delays in updating port list

Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  New
Status in Tempest:
  New

Bug description:
  When deleting an instance, the port list is not updated immediately.
  As a result - deleting net/subnet fails with error:

  409-{u'NeutronError': {u'message': u'Unable to complete operation on
  subnet UUID. One or more ports have an IP allocation from this
  subnet.', u'type': u'SubnetInUse', u'detail': u''}}

  (*) Happens only on automated scripts, since manual input isn't quick
  enough to catch this.

  (**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
  my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.

  (***) nova delete command returns BEFORE instance is deleted:
  > nova delete $instance_id ; nova show $instance_id
  will return the instance without failure

  (*****) might also affect Tempest during tearDown execution

  How to reproduce:
  **** script attached ****
  assumes:
  1. external network exists with floating ip range available ("public")
  2. image exists

  Setup:
  1 create network "private"
  2. create subnet
  3. create router:
  3.1 set router gateway to "public"
  3.2 set router interface to "private"
  4. create VM
  5. assign Floating IP to VM

  TearDown
  1. Delete / Disassociate Floating IP
  2. Delete VM
  3. detach router interface from subnet (router-interface-delete)
  4. Delete subnet/net

  Expected Result:
  subnet/net should be successfully deleted.

  Actual Results:
  "Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"

  409-{u'NeutronError': {u'message': u'Unable to complete operation on
  subnet UUID. One or more ports have an IP allocation from this
  subnet.', u'type': u'SubnetInUse', u'detail': u''}}

  script log:

  line 101 - VM port still in port list even though VM was deleted
  line 105 - subnet fails to delete
  line 117 - network successfully deleted after enough time passed for port list to update

    1 + EXT_NET_NAME=public
    2 + NET_NAME=my_net
    3 + SUBNET_NAME=my_subnet
    4 + ROUTER_NAME=my_router
    5 + SERVER_NAME=my_server
    6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
    7 + MASK=54.0.0
    8 + SERVER_IP=54.0.0.6
    9 ++ neutron net-list
   10 ++ grep public
   11 ++ awk '{print $2;}'
   12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
   13 ++ nova image-list
   14 ++ grep -w ' cirros-0.3.1-x86_64-uec  '
   15 ++ awk '{print $2;}'
   16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
   17 ++ grep -w id
   18 ++ neutron net-create my_net
   19 ++ awk '{print $4;}'
   20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
   21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
   22 ++ grep -w id
   23 ++ awk '{print $4;}'
   24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
   25 ++ neutron router-create my_router
   26 ++ awk '{print $4;}'
   27 ++ grep -w id
   28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
   29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
   30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
   31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
   32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
   33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
   34 +--------------------------------------+--------------------------------------+
   35 | Property                             | Value                                |
   36 +--------------------------------------+--------------------------------------+
   37 | OS-DCF:diskConfig                    | MANUAL                               |
   38 | OS-EXT-AZ:availability_zone          | nova                                 |
   39 | OS-EXT-STS:power_state               | 0                                    |
   40 | OS-EXT-STS:task_state                | scheduling                           |
   41 | OS-EXT-STS:vm_state                  | building                             |
   42 | OS-SRV-USG:launched_at               | None                                 |
   43 | OS-SRV-USG:terminated_at             | None                                 |
   44 | accessIPv4                           |                                      |
   45 | accessIPv6                           |                                      |
   46 | adminPass                            | 9VJ9XMrtFghW                         |
   47 | config_drive                         |                                      |
   48 | created                              | 2014-01-15T12:56:30Z                 |
   49 | flavor                               | m1.small                             |
   50 | hostId                               |                                      |
   51 | id                                   | d09bbee1-599c-4042-a409-a90e0ae80e3c |
   52 | image                                | cirros-0.3.1-x86_64-uec              |
   53 | key_name                             | None                                 |
   54 | metadata                             | {}                                   |
   55 | name                                 | my_server                            |
   56 | os-extended-volumes:volumes_attached | []                                   |
   57 | progress                             | 0                                    |
   58 | security_groups                      | [{"name": "default"}]                |
   59 | status                               | BUILD                                |
   60 | tenant_id                            | 66293febf7164c849b694a8d3f14cc1a     |
   61 | updated                              | 2014-01-15T12:56:30Z                 |
   62 | user_id                              | e895d5bf0dc64d44b707b9fbf5cbf6e0     |
   63 +--------------------------------------+--------------------------------------+
   64 ++ nova list
   65 ++ grep my_server
   66 ++ awk '{print $2;}'
   67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
   68 ++ neutron port-list
   69 ++ grep 54.0.0.6
   70 ++ awk '{print $2;}'
   71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
   72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
   73 ++ grep floating_ip_address
   74 ++ awk '{print $4;}'
   75 + FLOATING_IP=172.24.4.4
   76 ++ neutron floatingip-list
   77 ++ awk '{print $2;}'
   78 ++ grep 172.24.4.4
   79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
   80 + neutron port-list
   81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
   82 | id                                   | name | mac_address       | fixed_ips                                                                       |
   83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
   84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
   85 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
   86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
   87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
   88 | cafd4161-f840-4c87-a80b-71b0ef374b9e |      | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
   89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
   90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
   91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
   92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
   93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
   94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
   95 + neutron port-list
   96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
   97 | id                                   | name | mac_address       | fixed_ips                                                                       |
   98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
   99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
  100 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
  101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 |      | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
  102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
  103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
  105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
  106 + neutron port-list
  107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  108 | id                                   | name | mac_address       | fixed_ips                                                                       |
  109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 |      | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
  111 | 13630e5f-51c2-42b9-901a-2a208e5f986a |      | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
  112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e |      | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
  113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
  114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
  115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
  116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
  117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
  118 + set +x

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1269407/+subscriptions


Follow ups

References