yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1893618] Re: periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-master tempest test_shelve_unshelve_server failing in component-pipeline

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1893618@xxxxxxxxxxxxxxxxxx>
Date: Wed, 28 Apr 2021 09:06:25 -0000
Reply-to: Bug 1893618 <1893618@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
https://review.opendev.org/c/openstack/nova/+/749205 was the original
fix.

** Changed in: nova
       Status: New => Fix Released

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1893618

Title:
  periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-
  master tempest test_shelve_unshelve_server failing in component-
  pipeline

Status in OpenStack Compute (nova):
  Fix Released
Status in tripleo:
  Fix Released

Bug description:
  https://logserver.rdoproject.org/openstack-component-
  compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-
  centos-8-standalone-full-tempest-api-compute-
  master/b346467/logs/undercloud/var/log/tempest/stestr_results.html.gz

  traceback-1: {{{
  Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 66, in tearDown
      self.server_check_teardown()
    File "/usr/lib/python3.6/site-packages/tempest/api/compute/base.py", line 220, in server_check_teardown
      cls.server_id, 'ACTIVE')
    File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
      raise lib_exc.TimeoutException(message)
  tempest.lib.exceptions.TimeoutException: Request timed out
  Details: (ServerActionsTestJSON:tearDown) Server 41f15309-34bb-430d-8dad-7b9c8362a851 failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: SHELVED_OFFLOADED. Current task state: None.
  }}}

  traceback-2: {{{
  Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 649, in _unshelve_server
      server_info = self.client.show_server(self.server_id)['server']
    File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/servers_client.py", line 145, in show_server
      resp, body = self.get("servers/%s" % server_id)
    File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 313, in get
      return self.request('GET', url, extra_headers, headers)
    File "/usr/lib/python3.6/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
      method, url, extra_headers, headers, body, chunked)
    File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
      self._error_checker(resp, resp_body)
    File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 808, in _error_checker
      raise exceptions.NotFound(resp_body, resp=resp)
  tempest.lib.exceptions.NotFound: Object not found
  Details: {'code': 404, 'message': 'Instance None could not be found.'}
  }}}

  Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
      return f(*func_args, **func_kwargs)
    File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_server_actions.py", line 666, in test_shelve_unshelve_server
      waiters.wait_for_server_status(self.client, self.server_id, 'ACTIVE')
    File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
      raise lib_exc.TimeoutException(message)
  tempest.lib.exceptions.TimeoutException: Request timed out
  Details: (ServerActionsTestJSON:test_shelve_unshelve_server) Server 41f15309-34bb-430d-8dad-7b9c8362a851 failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: SHELVED_OFFLOADED. Current task state: None.


  Traceback in nova-compute logs https://logserver.rdoproject.org
  /openstack-component-compute/opendev.org/openstack/tripleo-ci/master
  /periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-
  master/b346467/logs/undercloud/var/log/containers/nova/nova-
  compute.log.1.gz:-

  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server [req-9280bac1-da23-4f45-b01c-b6012198d97e 10fe2caa6924408485c181adfc7377e8 df52aad2e4da4f07b4b7b4ff6644e121 - default default] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'encode'
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 78, in wrapped
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     raise value
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 189, in decorated_function
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     raise value
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 159, in decorated_function
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1440, in decorated_function
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 217, in decorated_function
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     raise value
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6539, in unshelve_instance
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     do_unshelve_instance()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6538, in do_unshelve_instance
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     request_spec)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6625, in _unshelve_instance
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self._nil_out_instance_obj_host_and_node(instance)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     raise value
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6609, in _unshelve_instance
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3672, in spawn
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4067, in _create_image
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     injection_info, fallback_from_host)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4198, in _create_and_inject_local_root
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     self._finalize_unshelve_qcow2_image(context, instance, backend)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 4218, in _finalize_unshelve_qcow2_image
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     root_fname = imagecache.get_cache_fname(base_image_ref)
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/imagecache.py", line 52, in get_cache_fname
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server     return hashlib.sha1(image_id.encode('utf-8')).hexdigest()
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'encode'
  2020-08-30 08:35:10.183 7 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1893618/+subscriptions