yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1548980] Re: nova list --deleted as admin fails with 404

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1548980@xxxxxxxxxxxxxxxxxx>
Date: Fri, 04 Mar 2016 22:16:58 -0000
Reply-to: Bug 1548980 <1548980@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/283820
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3d6bb233828ce63ae649e98e02dc59e04f3db2f5
Submitter: Jenkins
Branch:    master

commit 3d6bb233828ce63ae649e98e02dc59e04f3db2f5
Author: Matt Riedemann <mriedem@xxxxxxxxxx>
Date:   Tue Feb 23 16:34:39 2016 -0500

    Don't lazy-load instance.services if the instance is deleted
    
    The 2.16 microversion added the host_status extended
    server attribute which relies on the instance.services field.
    
    The primary join in the database for that field is dependent on
    the instance not being deleted.
    
    When listing deleted instances at microversion>=2.16, the
    compute API attempts to lazy-load the instance.services field
    which fails with an InstanceNotFound because the instance
    is deleted.
    
    In this case, it's best to just set instance.services to an
    empty ServiceList when lazy loading the services field on a
    deleted instance since the DB object won't have any value for
    the services attribute anyway.
    
    Change-Id: Ic2f239f634f917a5771b0401a5073546c710c036
    Closes-Bug: #1548980


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1548980

Title:
  nova list --deleted as admin fails with 404

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Mitaka devstack created about a week ago:

  stack@neutron:~/python-novaclient$ cd /opt/stack/nova
  stack@neutron:~/nova$ git log -1
  commit 11019fab7a2415cbca8b93e9346b21327e79688d
  Author: bhagyashris <bhagyashri.shewale@xxxxxxxxxxx>
  Date:   Tue Feb 16 01:13:23 2016 -0800

      Remove duplicate key from dictionary

      There is a duplicate dictionary key entry in test_vmops.py.
      Removed duplicate key 'display_name' from dictionary.

      TrivialFix

      Change-Id: I4e779bceb26077b95bd3ae4ab19e60152c126e34
  stack@neutron:~/nova$

  
  --

  I have a deleted instance:

  mysql> select id,uuid,display_name,deleted from nova.instances;
  +----+--------------------------------------+--------------+---------+
  | id | uuid                                 | display_name | deleted |
  +----+--------------------------------------+--------------+---------+
  |  1 | 55b9808b-7e01-44ba-ab84-c0bac34d57f1 | test1        |       1 |
  +----+--------------------------------------+--------------+---------+
  1 row in set (0.00 sec)

  
  I try to list deleted instances using 'nova list --deleted' and it fails with a 404.

  Checking the n-api logs there is an InstanceNotFound, it looks like
  when lazy-loading the instance.services field:

  2016-02-23 20:17:25.103 DEBUG nova.objects.instance [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Lazy-loading 'services' on Instance uuid 55b9808b-7e01-44ba-ab84-c0bac34d57f1 from (pid=17965) obj_load_attr /opt/stack/nova/nova/objects/instance.py:879
  2016-02-23 20:17:25.168 ERROR nova.api.openstack [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Caught error: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
  2016-02-23 20:17:25.168 TRACE nova.api.openstack Traceback (most recent call last):
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 140, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return req.get_response(self.application)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 457, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = req.get_response(self._app)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     application, catch_exc_info=False)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return resp(environ, start_response)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 672, in __call__
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     content_type, body, accept)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in _process_stack
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     request, action_args)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     **action_args)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/extended_server_attributes.py", line 90, in detail
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     instances.values())
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3461, in get_instances_host_statuses
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     host_status = self.get_instance_host_status(instance)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/api.py", line 3439, in get_instance_host_status
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     service = [service for service in instance.services if
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     self.obj_load_attr(name)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 902, in obj_load_attr
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     self._load_generic(attrname)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 746, in _load_generic
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     expected_attrs=[attrname])
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     result = fn(cls, context, *args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 399, in get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     use_slave=use_slave)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 283, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/api.py", line 670, in instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return f(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 709, in wrapper
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     return fn(*args, **kwargs)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1746, in instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     columns_to_join=columns_to_join)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1755, in _instance_get_by_uuid
  2016-02-23 20:17:25.168 TRACE nova.api.openstack     raise exception.InstanceNotFound(instance_id=uuid)
  2016-02-23 20:17:25.168 TRACE nova.api.openstack InstanceNotFound: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
  2016-02-23 20:17:25.168 TRACE nova.api.openstack

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1548980/+subscriptions
References

[Bug 1548980] [NEW] nova list --deleted as admin fails with 404
From: Matt Riedemann, 2016-02-23