yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1102886] Re: nova API returns 404 for floating-ip-list if an instance has disappeared

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Mon, 15 Sep 2014 12:08:52 -0000
Reply-to: Bug 1102886 <1102886@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Honestly, I'm pretty sure we managed to kill this bug. Please reopen if
you see it still crop up.

** Changed in: nova
       Status: Triaged => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1102886

Title:
  nova API returns 404 for floating-ip-list if an instance has
  disappeared

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  I'm not exactly sure how to reproduce the problem.
  In my case, one (out of two) compute nodes crashed, leaving a few instances listed which needed replacing.

  First the old instances was deleted, but it's still hanging around in the list since that compute node was crashed.
  Then released the floating ip via the horizon interface, and assigned it via nova client.

  While deleting and re-assigning floating ip's to new instances,
  something got set wrong. Which now result in floating ip's being
  broken for that tenant.

  ```
  nova --os-tenant-name API floating-ip-list
  ERROR: The resource could not be found. (HTTP 404) (Request-ID: req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c)
  ```

  The log from nova-api shows:

  ```
  2013-01-22 10:20:21 ERROR nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] Caught error: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
  2013-01-22 10:20:21 TRACE nova.api.openstack Traceback (most recent call last):
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 78, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
  2013-01-22 10:20:21 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
  2013-01-22 10:20:21 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
  2013-01-22 10:20:21 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 885, in __call__
  2013-01-22 10:20:21 TRACE nova.api.openstack     content_type, body, accept)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 933, in _process_stack
  2013-01-22 10:20:21 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1021, in dispatch
  2013-01-22 10:20:21 TRACE nova.api.openstack     return method(req=request, **action_args)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 158, in index
  2013-01-22 10:20:21 TRACE nova.api.openstack     self._set_metadata(context, floating_ip)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 130, in _set_metadata
  2013-01-22 10:20:21 TRACE nova.api.openstack     instance_uuid)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 118, in _get_instance
  2013-01-22 10:20:21 TRACE nova.api.openstack     return self.compute_api.get(context, instance_id)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1018, in get
  2013-01-22 10:20:21 TRACE nova.api.openstack     instance = self.db.instance_get_by_uuid(context, instance_id)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 576, in instance_get_by_uuid
  2013-01-22 10:20:21 TRACE nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 129, in wrapper
  2013-01-22 10:20:21 TRACE nova.api.openstack     return f(*args, **kwargs)
  2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1501, in instance_get_by_uuid
  2013-01-22 10:20:21 TRACE nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
  2013-01-22 10:20:21 TRACE nova.api.openstack InstanceNotFound: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
  2013-01-22 10:20:21 TRACE nova.api.openstack 
  2013-01-22 10:20:21 INFO nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] http://10.0.1.203:8774/v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips returned with HTTP 404
  2013-01-22 10:20:21 INFO nova.osapi_compute.wsgi.server [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] 10.0.1.50 - - [22/Jan/2013 10:20:21] "GET /v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips HTTP/1.1" 404 272 0.534521
  ```

  
  Maybe floating ip's assigned to non-existing instances can be reclaimed somehow without poking in the database?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1102886/+subscriptions