yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1518382] [NEW] nova list --deleted fails with a 500 response

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Fri, 20 Nov 2015 16:33:35 -0000
Reply-to: Bug 1518382 <1518382@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Nova version (mitaka):

stack@archive:/opt/stack/nova$ git log -1
commit f6a5a43e06c2af6325d7a3552c71e968565684fc
Merge: f268cf5 0df6fba
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Mon Nov 16 18:01:40 2015 +0000

    Merge "Remove duplicate server.kill on test shutdown"
stack@archive:/opt/stack/nova$


python-novaclient version:

stack@archive:/opt/stack/nova$ pip show python-novaclient
---
Metadata-Version: 2.0
Name: python-novaclient
Version: 2.35.0
Summary: Client library for OpenStack Compute API
Home-page: https://www.openstack.org
Author: OpenStack
Author-email: openstack-dev@xxxxxxxxxxxxxxxxxxx
License: Apache License, Version 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: oslo.i18n, oslo.serialization, python-keystoneclient, argparse, Babel, oslo.utils, iso8601, requests, pbr, six, PrettyTable, simplejson
stack@archive:/opt/stack/nova$


As a non-admin user, I created some servers and deleted them:

mysql> select display_name,uuid,deleted from nova.instances;
+--------------+--------------------------------------+---------+
| display_name | uuid                                 | deleted |
+--------------+--------------------------------------+---------+
| test         | ca3e57c0-cf37-40ab-9322-4cde6fe242d9 |       1 |
| test2        | 8d555e3b-41db-495d-8d03-4918a011472c |       2 |
| test3        | ef4c5911-958a-47e7-b836-42da9d32c209 |       3 |
+--------------+--------------------------------------+---------+
3 rows in set (0.00 sec)

As the admin user, I should be able to list them using 'nova list
--deleted', but that fails with a 500 because an InstanceNotFound is not
handled in the API code:

2015-11-20 16:26:11.033 ERROR nova.api.openstack.extensions [req-aaea6f78-bf6d-4c35-ad46-decba714767b admin demo] Unexpected exception in API method
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 280, in detail
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 406, in _get_servers
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     response = self._view_builder.detail(req, instance_list)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 151, in detail
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return self._list_view(self.show, request, instances, coll_name)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 163, in _list_view
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     server_list = [func(request, server)["server"] for server in servers]
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 291, in show
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     "flavor": self._get_flavor(request, instance),
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 223, in _get_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     instance_type = instance.get_flavor()
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 863, in get_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return getattr(self, attr)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 66, in getter
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     self.obj_load_attr(name)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 853, in obj_load_attr
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     self._load_flavor()
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 744, in _load_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 171, in wrapper
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 373, in get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/api.py", line 651, in instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1719, in instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1731, in _instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions InstanceNotFound: Instance ef4c5911-958a-47e7-b836-42da9d32c209 could not be found.
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions


It looks like it's trying to lazy load a flavor on the instance object which fails because it's not reading the deleted instance at that point.

** Affects: nova
     Importance: Medium
         Status: Triaged


** Tags: api db unified-objects

** Tags added: api db unified-objects

** Changed in: nova
       Status: New => Triaged

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1518382

Title:
  nova list --deleted fails with a 500 response

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Nova version (mitaka):

  stack@archive:/opt/stack/nova$ git log -1
  commit f6a5a43e06c2af6325d7a3552c71e968565684fc
  Merge: f268cf5 0df6fba
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Mon Nov 16 18:01:40 2015 +0000

      Merge "Remove duplicate server.kill on test shutdown"
  stack@archive:/opt/stack/nova$


  python-novaclient version:

  stack@archive:/opt/stack/nova$ pip show python-novaclient
  ---
  Metadata-Version: 2.0
  Name: python-novaclient
  Version: 2.35.0
  Summary: Client library for OpenStack Compute API
  Home-page: https://www.openstack.org
  Author: OpenStack
  Author-email: openstack-dev@xxxxxxxxxxxxxxxxxxx
  License: Apache License, Version 2.0
  Location: /usr/local/lib/python2.7/dist-packages
  Requires: oslo.i18n, oslo.serialization, python-keystoneclient, argparse, Babel, oslo.utils, iso8601, requests, pbr, six, PrettyTable, simplejson
  stack@archive:/opt/stack/nova$

  
  As a non-admin user, I created some servers and deleted them:

  mysql> select display_name,uuid,deleted from nova.instances;
  +--------------+--------------------------------------+---------+
  | display_name | uuid                                 | deleted |
  +--------------+--------------------------------------+---------+
  | test         | ca3e57c0-cf37-40ab-9322-4cde6fe242d9 |       1 |
  | test2        | 8d555e3b-41db-495d-8d03-4918a011472c |       2 |
  | test3        | ef4c5911-958a-47e7-b836-42da9d32c209 |       3 |
  +--------------+--------------------------------------+---------+
  3 rows in set (0.00 sec)

  As the admin user, I should be able to list them using 'nova list
  --deleted', but that fails with a 500 because an InstanceNotFound is
  not handled in the API code:

  2015-11-20 16:26:11.033 ERROR nova.api.openstack.extensions [req-aaea6f78-bf6d-4c35-ad46-decba714767b admin demo] Unexpected exception in API method
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions Traceback (most recent call last):
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 280, in detail
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 406, in _get_servers
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     response = self._view_builder.detail(req, instance_list)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 151, in detail
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return self._list_view(self.show, request, instances, coll_name)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 163, in _list_view
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     server_list = [func(request, server)["server"] for server in servers]
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 291, in show
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     "flavor": self._get_flavor(request, instance),
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 223, in _get_flavor
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     instance_type = instance.get_flavor()
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 863, in get_flavor
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return getattr(self, attr)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 66, in getter
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     self.obj_load_attr(name)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 853, in obj_load_attr
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     self._load_flavor()
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 744, in _load_flavor
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 171, in wrapper
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 373, in get_by_uuid
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     use_slave=use_slave)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/api.py", line 651, in instance_get_by_uuid
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1719, in instance_get_by_uuid
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1731, in _instance_get_by_uuid
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions InstanceNotFound: Instance ef4c5911-958a-47e7-b836-42da9d32c209 could not be found.
  2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions

  
  It looks like it's trying to lazy load a flavor on the instance object which fails because it's not reading the deleted instance at that point.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1518382/+subscriptions
Follow ups

[Bug 1518382] Re: nova list --deleted fails with a 500 response
From: Matt Riedemann, 2015-11-20