yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1696843] [NEW] metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 08 Jun 2017 19:42:45 -0000
Reply-to: Bug 1696843 <1696843@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_589590

Jun 08 18:36:06.549748 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: DEBUG nova.api.metadata.handler [-] Using cached metadata for instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f {{(pid=18770) get_metadata_by_instance_id /opt/stack/new/nova/nova/api/metadata/handler.py:78}}
Jun 08 18:36:06.589590 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: INFO nova.metadata.wsgi.server [None req-97c06652-908d-4603-a84d-40f818726b5d None None] Traceback (most recent call last):
Jun 08 18:36:06.589719 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 490, in handle_one_response
Jun 08 18:36:06.589804 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     result = self.application(self.environ, start_response)
Jun 08 18:36:06.589911 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
Jun 08 18:36:06.590006 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return app(environ, start_response)
Jun 08 18:36:06.590090 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Jun 08 18:36:06.590179 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     resp = self.call_func(req, *args, **self.kwargs)
Jun 08 18:36:06.590270 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Jun 08 18:36:06.590351 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return self.func(req, *args, **kwargs)
Jun 08 18:36:06.590428 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 125, in __call__
Jun 08 18:36:06.590508 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     response = req.get_response(self.application)
Jun 08 18:36:06.590585 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
Jun 08 18:36:06.590661 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     application, catch_exc_info=False)
Jun 08 18:36:06.590742 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
Jun 08 18:36:06.590824 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     app_iter = application(self.environ, start_response)
Jun 08 18:36:06.590906 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Jun 08 18:36:06.590983 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     resp = self.call_func(req, *args, **self.kwargs)
Jun 08 18:36:06.591060 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Jun 08 18:36:06.591136 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return self.func(req, *args, **kwargs)
Jun 08 18:36:06.591212 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/api/metadata/handler.py", line 121, in __call__
Jun 08 18:36:06.591287 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return data(req, meta_data)
Jun 08 18:36:06.591370 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/api/metadata/password.py", line 70, in handle_password
Jun 08 18:36:06.591446 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     instance = objects.Instance.get_by_uuid(ctxt, meta_data.uuid)
Jun 08 18:36:06.591531 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jun 08 18:36:06.591610 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     result = fn(cls, context, *args, **kwargs)
Jun 08 18:36:06.591687 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/objects/instance.py", line 463, in get_by_uuid
Jun 08 18:36:06.591763 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     use_slave=use_slave)
Jun 08 18:36:06.591839 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 236, in wrapper
Jun 08 18:36:06.591914 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(*args, **kwargs)
Jun 08 18:36:06.592000 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/objects/instance.py", line 455, in _db_instance_get_by_uuid
Jun 08 18:36:06.592069 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     columns_to_join=columns_to_join)
Jun 08 18:36:06.592159 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/api.py", line 745, in instance_get_by_uuid
Jun 08 18:36:06.592253 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
Jun 08 18:36:06.592339 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
Jun 08 18:36:06.592420 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(*args, **kwargs)
Jun 08 18:36:06.592501 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 281, in wrapped
Jun 08 18:36:06.592582 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(context, *args, **kwargs)
Jun 08 18:36:06.592664 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1921, in instance_get_by_uuid
Jun 08 18:36:06.592745 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     columns_to_join=columns_to_join)
Jun 08 18:36:06.593201 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1930, in _instance_get_by_uuid
Jun 08 18:36:06.593299 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     raise exception.InstanceNotFound(instance_id=uuid)
Jun 08 18:36:06.593384 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.
Jun 08 18:36:06.593487 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: : InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.
Jun 08 18:36:06.594103 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: INFO nova.metadata.wsgi.server [None req-97c06652-908d-4603-a84d-40f818726b5d None None] 10.1.0.11,10.40.184.140 "POST /openstack/2013-10-17/password HTTP/1.1" status: 500 len: 139 time: 0.0445061: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.

This was seen on this change:

https://review.openstack.org/#/c/436094/

The tempest test creates a server, then POSTs the password from the
guest to the metadata service, which should update the
instance.system_metadata with the password. Then Tempest calls the os-
server-password REST API to get the password and verify they are the
same, but it's not getting set during the metadata POST, and it looks
like because when the metadata handler is looking up the instance it's
not targeted to a cell.

** Affects: nova
     Importance: High
         Status: Triaged


** Tags: api cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1696843

Title:
  metadata: POST /openstack/2013-10-17/password fails with
  InstanceNotFound

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-
  neutron-full-ubuntu-
  xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_589590

  Jun 08 18:36:06.549748 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: DEBUG nova.api.metadata.handler [-] Using cached metadata for instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f {{(pid=18770) get_metadata_by_instance_id /opt/stack/new/nova/nova/api/metadata/handler.py:78}}
  Jun 08 18:36:06.589590 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: INFO nova.metadata.wsgi.server [None req-97c06652-908d-4603-a84d-40f818726b5d None None] Traceback (most recent call last):
  Jun 08 18:36:06.589719 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 490, in handle_one_response
  Jun 08 18:36:06.589804 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     result = self.application(self.environ, start_response)
  Jun 08 18:36:06.589911 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
  Jun 08 18:36:06.590006 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return app(environ, start_response)
  Jun 08 18:36:06.590090 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Jun 08 18:36:06.590179 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     resp = self.call_func(req, *args, **self.kwargs)
  Jun 08 18:36:06.590270 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Jun 08 18:36:06.590351 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return self.func(req, *args, **kwargs)
  Jun 08 18:36:06.590428 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 125, in __call__
  Jun 08 18:36:06.590508 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     response = req.get_response(self.application)
  Jun 08 18:36:06.590585 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
  Jun 08 18:36:06.590661 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     application, catch_exc_info=False)
  Jun 08 18:36:06.590742 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
  Jun 08 18:36:06.590824 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     app_iter = application(self.environ, start_response)
  Jun 08 18:36:06.590906 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Jun 08 18:36:06.590983 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     resp = self.call_func(req, *args, **self.kwargs)
  Jun 08 18:36:06.591060 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Jun 08 18:36:06.591136 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return self.func(req, *args, **kwargs)
  Jun 08 18:36:06.591212 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/api/metadata/handler.py", line 121, in __call__
  Jun 08 18:36:06.591287 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return data(req, meta_data)
  Jun 08 18:36:06.591370 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/api/metadata/password.py", line 70, in handle_password
  Jun 08 18:36:06.591446 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     instance = objects.Instance.get_by_uuid(ctxt, meta_data.uuid)
  Jun 08 18:36:06.591531 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Jun 08 18:36:06.591610 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     result = fn(cls, context, *args, **kwargs)
  Jun 08 18:36:06.591687 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/objects/instance.py", line 463, in get_by_uuid
  Jun 08 18:36:06.591763 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     use_slave=use_slave)
  Jun 08 18:36:06.591839 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 236, in wrapper
  Jun 08 18:36:06.591914 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(*args, **kwargs)
  Jun 08 18:36:06.592000 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/objects/instance.py", line 455, in _db_instance_get_by_uuid
  Jun 08 18:36:06.592069 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     columns_to_join=columns_to_join)
  Jun 08 18:36:06.592159 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/api.py", line 745, in instance_get_by_uuid
  Jun 08 18:36:06.592253 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  Jun 08 18:36:06.592339 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
  Jun 08 18:36:06.592420 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(*args, **kwargs)
  Jun 08 18:36:06.592501 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 281, in wrapped
  Jun 08 18:36:06.592582 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     return f(context, *args, **kwargs)
  Jun 08 18:36:06.592664 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1921, in instance_get_by_uuid
  Jun 08 18:36:06.592745 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     columns_to_join=columns_to_join)
  Jun 08 18:36:06.593201 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1930, in _instance_get_by_uuid
  Jun 08 18:36:06.593299 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]:     raise exception.InstanceNotFound(instance_id=uuid)
  Jun 08 18:36:06.593384 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.
  Jun 08 18:36:06.593487 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: : InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.
  Jun 08 18:36:06.594103 ubuntu-xenial-osic-cloud1-s3700-9216660 nova-api[18643]: INFO nova.metadata.wsgi.server [None req-97c06652-908d-4603-a84d-40f818726b5d None None] 10.1.0.11,10.40.184.140 "POST /openstack/2013-10-17/password HTTP/1.1" status: 500 len: 139 time: 0.0445061: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found.

  This was seen on this change:

  https://review.openstack.org/#/c/436094/

  The tempest test creates a server, then POSTs the password from the
  guest to the metadata service, which should update the
  instance.system_metadata with the password. Then Tempest calls the os-
  server-password REST API to get the password and verify they are the
  same, but it's not getting set during the metadata POST, and it looks
  like because when the metadata handler is looking up the instance it's
  not targeted to a cell.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1696843/+subscriptions
Follow ups

[Bug 1696843] Re: metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound
From: OpenStack Infra, 2017-06-09