yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1548980] [NEW] nova list --deleted as admin fails with 404

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Tue, 23 Feb 2016 20:22:38 -0000
Reply-to: Bug 1548980 <1548980@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Mitaka devstack created about a week ago:

stack@neutron:~/python-novaclient$ cd /opt/stack/nova
stack@neutron:~/nova$ git log -1
commit 11019fab7a2415cbca8b93e9346b21327e79688d
Author: bhagyashris <bhagyashri.shewale@xxxxxxxxxxx>
Date:   Tue Feb 16 01:13:23 2016 -0800

    Remove duplicate key from dictionary

    There is a duplicate dictionary key entry in test_vmops.py.
    Removed duplicate key 'display_name' from dictionary.

    TrivialFix

    Change-Id: I4e779bceb26077b95bd3ae4ab19e60152c126e34
stack@neutron:~/nova$


--

I have a deleted instance:

mysql> select id,uuid,display_name,deleted from nova.instances;
+----+--------------------------------------+--------------+---------+
| id | uuid                                 | display_name | deleted |
+----+--------------------------------------+--------------+---------+
|  1 | 55b9808b-7e01-44ba-ab84-c0bac34d57f1 | test1        |       1 |
+----+--------------------------------------+--------------+---------+
1 row in set (0.00 sec)


I try to list deleted instances using 'nova list --deleted' and it fails with a 404.

Checking the n-api logs there is an InstanceNotFound, it looks like when
lazy-loading the instance.services field:

2016-02-23 20:17:25.103 DEBUG nova.objects.instance [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Lazy-loading 'services' on Instance uuid 55b9808b-7e01-44ba-ab84-c0bac34d57f1 from (pid=17965) obj_load_attr /opt/stack/nova/nova/objects/instance.py:879
2016-02-23 20:17:25.168 ERROR nova.api.openstack [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Caught error: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
2016-02-23 20:17:25.168 TRACE nova.api.openstack Traceback (most recent call last):
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 140, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return req.get_response(self.application)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 457, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = req.get_response(self._app)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = self.app(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 672, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack     content_type, body, accept)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in _process_stack
2016-02-23 20:17:25.168 TRACE nova.api.openstack     request, action_args)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
2016-02-23 20:17:25.168 TRACE nova.api.openstack     **action_args)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/extended_server_attributes.py", line 90, in detail
2016-02-23 20:17:25.168 TRACE nova.api.openstack     instances.values())
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3461, in get_instances_host_statuses
2016-02-23 20:17:25.168 TRACE nova.api.openstack     host_status = self.get_instance_host_status(instance)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3439, in get_instance_host_status
2016-02-23 20:17:25.168 TRACE nova.api.openstack     service = [service for service in instance.services if
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2016-02-23 20:17:25.168 TRACE nova.api.openstack     self.obj_load_attr(name)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 902, in obj_load_attr
2016-02-23 20:17:25.168 TRACE nova.api.openstack     self._load_generic(attrname)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 746, in _load_generic
2016-02-23 20:17:25.168 TRACE nova.api.openstack     expected_attrs=[attrname])
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 399, in get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack     use_slave=use_slave)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 283, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/api.py", line 670, in instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 709, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack     return fn(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1746, in instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1755, in _instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
2016-02-23 20:17:25.168 TRACE nova.api.openstack InstanceNotFound: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
2016-02-23 20:17:25.168 TRACE nova.api.openstack

** Affects: nova
     Importance: High
         Status: Triaged


** Tags: api

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1548980

Title:
  nova list --deleted as admin fails with 404

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Mitaka devstack created about a week ago:

  stack@neutron:~/python-novaclient$ cd /opt/stack/nova
  stack@neutron:~/nova$ git log -1
  commit 11019fab7a2415cbca8b93e9346b21327e79688d
  Author: bhagyashris <bhagyashri.shewale@xxxxxxxxxxx>
  Date:   Tue Feb 16 01:13:23 2016 -0800

      Remove duplicate key from dictionary

      There is a duplicate dictionary key entry in test_vmops.py.
      Removed duplicate key 'display_name' from dictionary.

      TrivialFix

      Change-Id: I4e779bceb26077b95bd3ae4ab19e60152c126e34
  stack@neutron:~/nova$

  
  --

  I have a deleted instance:

  mysql> select id,uuid,display_name,deleted from nova.instances;
  +----+--------------------------------------+--------------+---------+
  | id | uuid                                 | display_name | deleted |
  +----+--------------------------------------+--------------+---------+
  |  1 | 55b9808b-7e01-44ba-ab84-c0bac34d57f1 | test1        |       1 |
  +----+--------------------------------------+--------------+---------+
  1 row in set (0.00 sec)

  
  I try to list deleted instances using 'nova list --deleted' and it fails with a 404.

  Checking the n-api logs there is an InstanceNotFound, it looks like
  when lazy-loading the instance.services field:

  2016-02-23 20:17:25.103 DEBUG nova.objects.instance [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Lazy-loading 'services' on Instance uuid 55b9808b-7e01-44ba-ab84-c0bac34d57f1 from (pid=17965) obj_load_attr /opt/stack/nova/nova/objects/instance.py:879
  2016-02-23 20:17:25.168 ERROR nova.api.openstack [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Caught error: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
  2016-02-23 20:17:25.168 TRACE nova.api.openstack Traceback (most recent call last):
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 140, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return req.get_response(self.application)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 457, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = req.get_response(self._app)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 672, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     content_type, body, accept)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in _process_stack
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     request, action_args)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     **action_args)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/extended_server_attributes.py", line 90, in detail
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     instances.values())
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3461, in get_instances_host_statuses
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     host_status = self.get_instance_host_status(instance)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3439, in get_instance_host_status
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     service = [service for service in instance.services if
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     self.obj_load_attr(name)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 902, in obj_load_attr
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     self._load_generic(attrname)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 746, in _load_generic
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     expected_attrs=[attrname])
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 399, in get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     use_slave=use_slave)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 283, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/api.py", line 670, in instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 709, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return fn(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1746, in instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1755, in _instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack InstanceNotFound: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
  2016-02-23 20:17:25.168 TRACE nova.api.openstack

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1548980/+subscriptions
Follow ups

[Bug 1548980] Re: nova list --deleted as admin fails with 404
From: OpenStack Infra, 2016-03-04