yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1627177] Re: Liberty to Mitaka nova list fails when force create a VM on a bad compute node that is not reachable

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Fri, 09 Dec 2016 15:19:30 -0000
Reply-to: Bug 1627177 <1627177@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
As listed, this is an issue with memcache needing to be flushed after
upgrade. This should be handled at the orchestration layer for upgrade.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1627177

Title:
  Liberty to Mitaka nova list fails when force create a VM on a bad
  compute node that is not reachable

Status in OpenStack Compute (nova):
  Invalid
Status in openstack-ansible:
  Invalid

Bug description:
  Liberty to Mitaka Upgrade

  Nova list fails when a user forces a compute creation on one of the
  nova computes that is in a bad state.

  nova list limiting to the other hypervisors works though and all other
  api commands work, except for the nova list.

  This is happening when u create a vm on a bad compute else it works
  fine.

  
  Log 

  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack [req-16a27bf2-bebe-4f13-a408-4b5b09129d6c 111ea8c6602e44bc8d7b9a125c86f12a 48d9424cadf145e59c98d5ca53c54f11 - - -] Caught error: 'str' object has no attribute 'metadata'
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack Traceback (most recent call last):
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 139, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return req.get_response(self.application)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/request.py", line 1317, in send
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     response = req.get_response(self._app)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/request.py", line 1317, in send
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     application, catch_exc_info=False)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/routes/middleware.py", line 136, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     response = self.app(environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return resp(environ, start_response)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 672, in __call__
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     content_type, body, accept)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     request, action_args)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     **action_args)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/compute/extended_availability_zone.py", line 51, in detail
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     self._extend_server(context, server, db_instance)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/api/openstack/compute/extended_availability_zone.py", line 33, in _extend_server
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     az = avail_zone.get_instance_availability_zone(context, instance)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/availability_zones.py", line 169, in get_instance_availability_zone
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     az = cache.get(cache_key)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/nova/cache_utils.py", line 151, in get
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     value = self.region.get(key)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/dogpile/cache/region.py", line 456, in get
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     expiration_time, ignore_expiration)(value)
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack   File "/openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/dogpile/cache/region.py", line 475, in value_fn
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack     current_time - value.metadata["ct"] > expiration_time:
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack AttributeError: 'str' object has no attribute 'metadata'
  2016-09-23 12:51:24.479 2241 ERROR nova.api.openstack
  2016-09-23 12:51:24.480 2241 INFO nova.api.openstack [req-16a27bf2-bebe-4f13-a408-4b5b09129d6c 111ea8c6602e44bc8d7b9a125c86f12a 48d9424cadf145e59c98d5ca53c54f11 - - -] http://xxxxxxxxxxxxxxxxxxxxx:8774/v2.1/48d9424cadf145e59c98d5ca53c54f11/servers/detail returned with HTTP 500

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1627177/+subscriptions