yahoo-eng-team team mailing list archive

[OpenStack Infra <1946339@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/814036
Committed: https://opendev.org/openstack/nova/commit/61fc81a6761d34afdfc4a6d1c4c953802fd8a179
Submitter: "Zuul (22348)"
Branch:    master

commit 61fc81a6761d34afdfc4a6d1c4c953802fd8a179
Author: Balazs Gibizer <balazs.gibizer@xxxxxxxx>
Date:   Thu Oct 14 18:09:18 2021 +0200

    Prevent leaked eventlets to send notifications
    
    In out functional tests we run nova services as eventlets. Also those
    services can spawn there own eventlets for RPC or other parallel
    processing. The test case executor only sees and tracks the main
    eventlet where the code of the test case is running. When that is
    finishes the test executor considers the test case to be finished
    regardless of the other spawned eventlets. This could lead to leaked
    eventlets that are running in parallel with later test cases.
    
    One way that it can cause trouble is via the global variables in
    nova.rpc module. Those globals are re-initialized for each test case so
    they are not directly leaking information between test cases. However if
    a late eventlet calls nova.rpc.get_versioned_notifier() it will get a
    totally usable FakeVersionedNotifier object regardless of which test
    case this notifier is belongs to or which test case the eventlet belongs
    to. This way the late eventlet can send a notification to the currently
    running test case and therefore can make it fail.
    
    The current case we saw is the following:
    
    1) The test case
      nova.tests.functional.test_servers.ServersTestV219.test_description_errors
      creates a server but don't wait for it to reach terminal state (ACTIVE
      / ERROR). This test case finishes quickly but leaks running eventlets
      in the background waiting for some RPC call to return.
    2) As the test case finished the cleanup code deletes the test case
       specific setup, including the DB.
    3) The test executor moves forward and starts running another test case
    4) 60 seconds later the leaked eventlet times out waiting for the RPC
       call to return and tries doing things, but fails as the DB is already
       gone. Then it tries to  report this as an error notification. It calls
       nova.rpc.get_versioned_notifier() and gets a fresh notifier that is
       connected to the currently running test case. Then emits the error
       notification there.
    5) The currently running test case also waits for an error notification
       to be triggered by the currently running test code. But it gets the
       notification form the late eventlet first. As the content of the
       notification does not match with the expectations the currently
       running test case fails. The late eventlet prints a lot of
       error about the DB being gone making the troubleshooting pretty hard.
    
    This patch proposes a way to fix this by marking each eventlet at spawn
    time with the id of the test case that was directly or indirectly
    started it.
    
    Then when the NotificationFixture gets a notification it compares the
    test case id stored in the calling eventlet with the id of the test case
    initialized the NotificationFixture. If the two ids do not match then
    the fixture ignores the notification and raises an exception to the
    caller eventlet to make it terminate.
    
    Change-Id: I012dcf63306bae624dc4f66aae6c6d96a20d4327
    Closes-Bug: #1946339


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1946339

Title:
  test_unshelve_offloaded_server_with_qos_port_pci_update_fails

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4d3/812392/2/check/nova-
  tox-functional-py38/4d32a96/testr_results.html

  2021-10-07 00:06:06,980 ERROR [nova.compute.manager] Instance failed to spawn
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 6651, in _unshelve_instance
      update(
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1531, in update_pci_request_spec_with_allocated_interface_name
      raise ex(
  nova.exception.UnexpectedResourceProviderNameForPCIRequest: Resource provider 125ffd8a-3163-425a-b397-f915c602b3eb used to allocate resources for the pci request 7059503b-a648-40fd-a561-5ca769304bee does not have a properly formatted name. Expected name format is <hostname>:<agentname>:<interfacename>, but got invalid-device-rp-name
  2021-10-07 00:06:06,994 INFO [placement.requestlog] 127.0.0.1 "DELETE /placement/allocations/718d8d5b-8d34-4c11-9080-1952cc824086" status: 204 len: 0 microversion: 1.0
  2021-10-07 00:06:06,995 INFO [nova.scheduler.client.report] Deleted allocations for instance 718d8d5b-8d34-4c11-9080-1952cc824086
  2021-10-07 00:06:07,029 INFO [nova.compute.manager] Successfully reverted task state from spawning on failure for instance.
  2021-10-07 00:06:07,052 ERROR [oslo_messaging.rpc.server] Exception during message handling
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      result = func(ctxt, **new_args)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 71, in wrapped
      _emit_versioned_exception_notification(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 63, in wrapped
      return f(self, context, *args, **kw)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 183, in decorated_function
      LOG.warning("Failed to revert task state for instance. "
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 154, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1433, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 211, in decorated_function
      compute_utils.add_instance_fault_from_exc(context,
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 6595, in unshelve_instance
      do_unshelve_instance()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 6592, in do_unshelve_instance
      self._unshelve_instance(
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 6697, in _unshelve_instance
      self._nil_out_instance_obj_host_and_node(instance)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 6651, in _unshelve_instance
      update(
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1531, in update_pci_request_spec_with_allocated_interface_name
      raise ex(
  nova.exception.UnexpectedResourceProviderNameForPCIRequest: Resource provider 125ffd8a-3163-425a-b397-f915c602b3eb used to allocate resources for the pci request 7059503b-a648-40fd-a561-5ca769304bee does not have a properly formatted name. Expected name format is <hostname>:<agentname>:<interfacename>, but got invalid-device-rp-name
  2021-10-07 00:06:07,183 INFO [oslo_db.sqlalchemy.provision] DROP ALL OBJECTS, BACKEND sqlite://
  2021-10-07 00:06:07,201 INFO [oslo_db.sqlalchemy.provision] DISPOSE ENGINE Engine(sqlite://)
  }}}

  stderr: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send
      reply, failure = reply_q.get(timeout=timeout)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get
      return waiter.wait()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait
      return get_hub().switch()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
      return self.greenlet.switch()
  _queue.Empty

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 200, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2263, in _do_build_and_run_instance
      self.compute_task_api.build_instances(context, [instance],
    File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/api.py", line 119, in build_instances
      self.conductor_compute_rpcapi.build_instances(context,
    File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/rpcapi.py", line 390, in build_instances
      cctxt.cast(context, 'build_instances', **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/fixtures/_fixtures/monkeypatch.py", line 89, in avoid_get
      return captured_method(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call
      result = self.transport._send(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
      return self._driver.send(target, ctxt, message,
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
      return self._send(target, ctxt, message, wait_for_reply, timeout,
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send
      raise oslo_messaging.MessagingTimeout(
  oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
      cursor.execute(statement, parameters)
  sqlite3.OperationalError: no such table: instance_faults

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1433, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 211, in decorated_function
      compute_utils.add_instance_fault_from_exc(context,
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 153, in add_instance_fault_from_exc
      fault_obj.create()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
      return fn(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/instance_fault.py", line 76, in create
      db_fault = db.instance_fault_create(self._context, values)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 190, in wrapper
      return f(context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 3881, in instance_fault_create
      fault_ref.save(context.session)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_db/sqlalchemy/models.py", line 48, in save
      session.flush()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3339, in flush
      self._flush(objects)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3479, in _flush
      transaction.rollback(_capture_exception=True)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
      compat.raise_(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3439, in _flush
      flush_context.execute()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
      rec.execute(self)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
      util.preloaded.orm_persistence.save_obj(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj
      _emit_insert_statements(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements
      result = connection._execute_20(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1611, in _execute_20
      return meth(self, args_10style, kwargs_10style, execution_options)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
      return connection._execute_clauseelement(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
      ret = self._execute_context(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
      self._handle_dbapi_exception(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2021, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
      cursor.execute(statement, parameters)
  oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: instance_faults
  [SQL: INSERT INTO instance_faults (created_at, updated_at, deleted_at, deleted, instance_uuid, code, message, details, host) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)]
  [parameters: ('2021-10-07 00:06:06.747150', None, None, 0, '5903406a-6fff-4179-99bf-8bab9c33f0cb', 500, 'MessagingTimeout', 'Traceback (most recent call last):\n  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging ... (2344 characters truncated) ... l_fake.py", line 213, in _send\n    raise oslo_messaging.MessagingTimeout(\noslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\n', 'host2')]
  (Background on this error at: https://sqlalche.me/e/14/e3q8)

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
      cursor.execute(statement, parameters)
  sqlite3.OperationalError: no such table: instance_actions

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
      timer()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/utils.py", line 680, in context_wrapper
      func(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2115, in _locked_do_build_and_run_instance
      result = self._do_build_and_run_instance(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 71, in wrapped
      _emit_versioned_exception_notification(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 63, in wrapped
      return f(self, context, *args, **kw)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 183, in decorated_function
      LOG.warning("Failed to revert task state for instance. "
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 154, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1433, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1394, in __exit__
      objects.InstanceActionEvent.event_finish_with_failure(
    File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/base.py", line 355, in wrapper
      return fn.__get__(None, obj)(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
      result = fn(cls, context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/objects/instance_action.py", line 227, in event_finish_with_failure
      db_event = db.action_event_finish(context, values)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
      ectxt.value = e.inner_exc
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 190, in wrapper
      return f(context, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 4078, in action_event_finish
      action = _action_get_by_request_id(context, values['instance_uuid'],
    File "/home/zuul/src/opendev.org/openstack/nova/nova/db/main/api.py", line 4014, in _action_get_by_request_id
      result = model_query(context, models.InstanceAction).\
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2762, in first
      return self.limit(1)._iter().first()
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2846, in _iter
      result = self.session.execute(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
      result = conn._execute_20(statement, params or {}, execution_options)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1611, in _execute_20
      return meth(self, args_10style, kwargs_10style, execution_options)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
      return connection._execute_clauseelement(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
      ret = self._execute_context(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
      self._handle_dbapi_exception(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2021, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
      self.dialect.do_execute(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
      cursor.execute(statement, parameters)
  oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: instance_actions
  [SQL: SELECT instance_actions.created_at AS instance_actions_created_at, instance_actions.updated_at AS instance_actions_updated_at, instance_actions.deleted_at AS instance_actions_deleted_at, instance_actions.deleted AS instance_actions_deleted, instance_actions.id AS instance_actions_id, instance_actions.action AS instance_actions_action, instance_actions.instance_uuid AS instance_actions_instance_uuid, instance_actions.request_id AS instance_actions_request_id, instance_actions.user_id AS instance_actions_user_id, instance_actions.project_id AS instance_actions_project_id, instance_actions.start_time AS instance_actions_start_time, instance_actions.finish_time AS instance_actions_finish_time, instance_actions.message AS instance_actions_message 
  FROM instance_actions 
  WHERE instance_actions.deleted = ? AND instance_actions.instance_uuid = ? AND instance_actions.request_id = ? ORDER BY instance_actions.created_at DESC, instance_actions.id DESC
   LIMIT ? OFFSET ?]
  [parameters: (0, '5903406a-6fff-4179-99bf-8bab9c33f0cb', 'req-14ab570f-ce9a-4a2e-a753-db18915fbe14', 1, 0)]
  (Background on this error at: https://sqlalche.me/e/14/e3q8)
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/test_servers_resource_request.py", line 2491, in test_unshelve_offloaded_server_with_qos_port_pci_update_fails
      self.assertEqual(
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/testtools/testcase.py", line 393, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: 'UnexpectedResourceProviderNameForPCIRequest' != 'DBNonExistentTable'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1946339/+subscriptions