yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1739782] [NEW] openstack usage list throws API error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: bjolo <bjorn.lofdahl@xxxxxxxxx>
Date: Fri, 22 Dec 2017 14:23:22 -0000
Reply-to: Bug 1739782 <1739782@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

env: kolla-ansible newton release

after archiving 10.000.000 deleted rows, openstack usage list throws
error

steps done
> openstack usage list -> works and gives correct output
> nova-manage db archive_deleted_rows --max-rows 500000
command above executed multiple times, but not all deleted rows have been archived. There are more rows that should be archived
> openstack usage list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'> (HTTP 500) (Request-ID: req-e6ef0ddb-8377-4974-b342-61bbfc7dc70c)


from the nova-api log

2017-12-22 15:05:03.151 36 INFO nova.metadata.wsgi.server [-] 136.225.154.5,136.225.154.7 "GET /latest/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 131 time: 0.0058219
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] Unexpected exception in API method
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     detailed=detailed)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     info['memory_mb'] = instance.flavor.memory_mb
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     self._load_flavor()
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     use_slave=use_slave)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions InstanceNotFound: Instance d9e7610b-d154-4902-9a72-7d4d3171586a could not be found.
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions
2017-12-22 15:05:30.735 21 INFO nova.api.openstack.wsgi [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'>

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1739782

Title:
  openstack usage list throws API error

Status in OpenStack Compute (nova):
  New

Bug description:
  env: kolla-ansible newton release

  after archiving 10.000.000 deleted rows, openstack usage list throws
  error

  steps done
  > openstack usage list -> works and gives correct output
  > nova-manage db archive_deleted_rows --max-rows 500000
  command above executed multiple times, but not all deleted rows have been archived. There are more rows that should be archived
  > openstack usage list
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceNotFound'> (HTTP 500) (Request-ID: req-e6ef0ddb-8377-4974-b342-61bbfc7dc70c)

  
  from the nova-api log

  2017-12-22 15:05:03.151 36 INFO nova.metadata.wsgi.server [-] 136.225.154.5,136.225.154.7 "GET /latest/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 131 time: 0.0058219
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] Unexpected exception in API method
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     detailed=detailed)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     info['memory_mb'] = instance.flavor.memory_mb
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     self.obj_load_attr(name)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     self._load_flavor()
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     expected_attrs=['flavor', 'system_metadata'])
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions     raise exception.InstanceNotFound(instance_id=uuid)
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions InstanceNotFound: Instance d9e7610b-d154-4902-9a72-7d4d3171586a could not be found.
  2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions
  2017-12-22 15:05:30.735 21 INFO nova.api.openstack.wsgi [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceNotFound'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1739782/+subscriptions
Follow ups

[Bug 1739782] Re: openstack usage list throws API error
From: Launchpad Bug Tracker, 2018-02-24