yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1102886@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

I'm not exactly sure how to reproduce the problem.
In my case, one (out of two) compute nodes crashed, leaving a few instances listed which needed replacing.

First the old instances was deleted, but it's still hanging around in the list since that compute node was crashed.
Then released the floating ip via the horizon interface, and assigned it via nova client.

While deleting and re-assigning floating ip's to new instances,
something got set wrong. Which now result in floating ip's being broken
for that tenant.

```
nova --os-tenant-name API floating-ip-list
ERROR: The resource could not be found. (HTTP 404) (Request-ID: req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c)
```

The log from nova-api shows:

```
2013-01-22 10:20:21 ERROR nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] Caught error: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
2013-01-22 10:20:21 TRACE nova.api.openstack Traceback (most recent call last):
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 78, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return req.get_response(self.application)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2013-01-22 10:20:21 TRACE nova.api.openstack     application, catch_exc_info=False)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2013-01-22 10:20:21 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return self.app(env, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     response = self.app(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2013-01-22 10:20:21 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 885, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack     content_type, body, accept)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 933, in _process_stack
2013-01-22 10:20:21 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1021, in dispatch
2013-01-22 10:20:21 TRACE nova.api.openstack     return method(req=request, **action_args)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 158, in index
2013-01-22 10:20:21 TRACE nova.api.openstack     self._set_metadata(context, floating_ip)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 130, in _set_metadata
2013-01-22 10:20:21 TRACE nova.api.openstack     instance_uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 118, in _get_instance
2013-01-22 10:20:21 TRACE nova.api.openstack     return self.compute_api.get(context, instance_id)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1018, in get
2013-01-22 10:20:21 TRACE nova.api.openstack     instance = self.db.instance_get_by_uuid(context, instance_id)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 576, in instance_get_by_uuid
2013-01-22 10:20:21 TRACE nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 129, in wrapper
2013-01-22 10:20:21 TRACE nova.api.openstack     return f(*args, **kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1501, in instance_get_by_uuid
2013-01-22 10:20:21 TRACE nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack InstanceNotFound: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
2013-01-22 10:20:21 TRACE nova.api.openstack 
2013-01-22 10:20:21 INFO nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] http://10.0.1.203:8774/v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips returned with HTTP 404
2013-01-22 10:20:21 INFO nova.osapi_compute.wsgi.server [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] 10.0.1.50 - - [22/Jan/2013 10:20:21] "GET /v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips HTTP/1.1" 404 272 0.534521
```


Maybe floating ip's assigned to non-existing instances can be reclaimed somehow without poking in the database?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
nova API returns 404 for floating-ip-list if an instance has disappeared
https://bugs.launchpad.net/bugs/1102886
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova).