yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1610153] [NEW] nova list can sometimes return 404

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date: Fri, 05 Aug 2016 09:35:42 -0000
Reply-to: Bug 1610153 <1610153@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On a large number of instances 'nova list' may return 404, probably this
is because some instances are deleted during command execution. Trace:

2016-08-05 09:30:52.666 878 ERROR nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] Caught error: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found.
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack Traceback (most recent call last):
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 139, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return req.get_response(self.application)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     application, catch_exc_info=False)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     response = req.get_response(self._app)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     application, catch_exc_info=False)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     response = self.app(environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 672, in __call__
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     content_type, body, accept)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     request, action_args)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     **action_args)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/extended_server_attributes.py", line 97, in detail
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     instances.values())
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3597, in get_instances_host_statuses
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     host_status = self.get_instance_host_status(instance)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3575, in get_instance_host_status
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     service = [service for service in instance.services if
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     self.obj_load_attr(name)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 925, in obj_load_attr
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     self._load_generic(attrname)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 753, in _load_generic
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     expected_attrs=[attrname])
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     result = fn(cls, context, *args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 399, in get_by_uuid
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     use_slave=use_slave)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 285, in wrapper
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(*args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     columns_to_join=columns_to_join)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 696, in instance_get_by_uuid
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(*args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 330, in wrapped
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(context, *args, **kwargs)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1967, in instance_get_by_uuid
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     columns_to_join=columns_to_join)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1976, in _instance_get_by_uuid
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack InstanceNotFound: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found.
2016-08-05 09:30:52.666 878 ERROR nova.api.openstack
2016-08-05 09:30:52.672 878 INFO nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] http://192.168.0.2:8774/v2.1/servers/detail?all_tenants=1 returned with HTTP 404
2016-08-05 09:30:52.679 878 INFO nova.osapi_compute.wsgi.server [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] 192.168.0.2 "GET /v2.1/servers/detail?all_tenants=1 HTTP/1.1" status: 404 len: 363 time: 7.7833738

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1610153

Title:
  nova list can sometimes return 404

Status in OpenStack Compute (nova):
  New

Bug description:
  On a large number of instances 'nova list' may return 404, probably
  this is because some instances are deleted during command execution.
  Trace:

  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] Caught error: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found.
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack Traceback (most recent call last):
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 139, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return req.get_response(self.application)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     response = req.get_response(self._app)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     response = self.app(environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 672, in __call__
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     content_type, body, accept)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     request, action_args)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     **action_args)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/extended_server_attributes.py", line 97, in detail
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     instances.values())
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3597, in get_instances_host_statuses
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     host_status = self.get_instance_host_status(instance)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3575, in get_instance_host_status
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     service = [service for service in instance.services if
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     self.obj_load_attr(name)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 925, in obj_load_attr
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     self._load_generic(attrname)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 753, in _load_generic
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     expected_attrs=[attrname])
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     result = fn(cls, context, *args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 399, in get_by_uuid
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     use_slave=use_slave)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 285, in wrapper
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(*args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     columns_to_join=columns_to_join)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 696, in instance_get_by_uuid
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(*args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 330, in wrapped
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     return f(context, *args, **kwargs)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1967, in instance_get_by_uuid
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     columns_to_join=columns_to_join)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1976, in _instance_get_by_uuid
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack InstanceNotFound: Instance 28c33ed4-c1a4-432c-96de-059b94a3dd91 could not be found.
  2016-08-05 09:30:52.666 878 ERROR nova.api.openstack
  2016-08-05 09:30:52.672 878 INFO nova.api.openstack [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] http://192.168.0.2:8774/v2.1/servers/detail?all_tenants=1 returned with HTTP 404
  2016-08-05 09:30:52.679 878 INFO nova.osapi_compute.wsgi.server [req-707a0e40-67cf-43a9-865d-c44a678b2986 2e2a43e956f344d184e40771d59c991d 13f508a4dd0e4b538561be2afcf5d699 - - -] 192.168.0.2 "GET /v2.1/servers/detail?all_tenants=1 HTTP/1.1" status: 404 len: 363 time: 7.7833738

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1610153/+subscriptions