yahoo-eng-team team mailing list archive

[Marcus Klein <kleini76@xxxxxxxxx>]

Okay, we will do it that way.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975

Title:
  Nova API floating IP delete fails (race with concurrent server delete)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  This happens with OpenStack Newton.
  We use some outdated openstack clients, that do not provide the "floating ip delete" command. Therefore we use the "ip floating delete" command.

  The floating IP is allocated using the "ip floating create" command. The floating IP is then associated to a newly started virtual machine.
  After some automated tests, the command to delete the virtual machine is sent. To keep the project clean, the floating IP is deleted afterwards using the "ip floating delete" command. This command fails from time to time due to some race condition.

  This race condition seems to be the following. Nova API requests
  information about the floating IP from neutron:

  GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
  Host: controller:9696
  Connection: keep-alive
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-neutronclient
  X-Auth-Token: gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8

  {"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
  "status": "ACTIVE", "description": "", "updated_at":
  "2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
  "f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
  "192.168.3.10", "floating_ip_address": "10.50.0.62",
  "revision_number": 2, "port_id":
  "4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
  ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
  "2017-07-18T03:37:04Z", "tenant_id":
  "7829521236b143d2a6778e09ba588ec0", "project_id":
  "7829521236b143d2a6778e09ba588ec0"}}

  In the response the floating IP still seems to be associated to the
  virtual machine - but the request to delete the virtual machine was
  already sent.

  Nova API requests information about the port, too:

  GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
  Host: controller:9696
  Connection: keep-alive
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-neutronclient
  X-Auth-Token: gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8

  {"NeutronError": {"message": "Port
  4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
  "PortNotFound", "detail": ""}}

  But the virtual machine and its floating IP association seems to be
  deleted in the meantime. This makes Nova API fail to delete the
  floating IP:

  2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi [req-b8f7414b-6398-4738-b660-fac693a187ab 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
  2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server [req-b8f7414b-6398-4738-b660-fac693a187ab 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 404 len: 466 time: 0.2575810

  openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6

  2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server [req-dafa9bec-fe87-47f6-8c44-4420b074da4d 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 200 len: 475 time: 0.2409530
  2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server [req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931 807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173 7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141 27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE /v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6 HTTP/1.1" status: 202 len: 337 time: 0.7844548

  GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
  Host: controller:9696
  Connection: keep-alive
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-neutronclient
  X-Auth-Token: gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A

  {"floatingip": {"router_id": null, "status": "DOWN", "description":
  "", "updated_at": "2017-07-18T03:38:41Z", "dns_domain": "",
  "floating_network_id": "f3d64d76-a4e5-473b-878f-78b032ab89df",
  "fixed_ip_address": null, "floating_ip_address": "10.50.0.62",
  "revision_number": 3, "port_id": null, "id": "3aeb2a7e-
  ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
  "2017-07-18T03:37:04Z", "tenant_id":
  "7829521236b143d2a6778e09ba588ec0", "project_id":
  "7829521236b143d2a6778e09ba588ec0"}}

  - some other requests to networks and the IP address 10.50.0.62

  DELETE /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
  Host: controller:9696
  Connection: keep-alive
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-neutronclient
  X-Auth-Token: gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A
  Content-Length: 0

  HTTP/1.1 204 No Content
  Content-Length: 0
  X-Openstack-Request-Id: req-5aee2081-b069-42c7-9c21-57b58bcb6fc5
  Date: Tue, 18 Jul 2017 09:32:51 GMT
  Connection: keep-alive

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1704975/+subscriptions