yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1882608] Fix included in openstack/nova train-eol

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1882608@xxxxxxxxxxxxxxxxxx>
Date: Thu, 31 Aug 2023 13:44:16 -0000
Reply-to: Bug 1882608 <1882608@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This issue was fixed in the openstack/nova train-eol  release.

** Changed in: nova/train
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1882608

Title:
  DELETE fails with HTTP 500, StaleDataError: UPDATE statement on table
  'instance_mappings' expected to update 1 row(s); 0 were matched

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) train series:
  Fix Released
Status in OpenStack Compute (nova) ussuri series:
  Fix Released

Bug description:
  Noticed in a failed nova-grenade-multinode gate job where a resource
  cleanup (server delete) during a ServersNegativeTestJSON test results
  in a 500 error and the job fails with:

  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi [None req-ab8b5ad1-c168-4f7e-9bfc-42b202b9894f tempest-ServersNegativeTestJSON-1435542876 tempest-ServersNegativeTestJSON-1435542876] Unexpected exception in API method: sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'instance_mappings' expected to update 1 row(s); 0 were matched.
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 671, in wrapped
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 990, in delete
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._delete(req.environ['nova.context'], req, id)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 798, in _delete
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self.compute_api.delete(context, instance)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 224, in inner
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 151, in inner
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 2479, in delete
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._delete_instance(context, instance)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 2471, in _delete_instance
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     task_state=task_states.DELETING)
  Jun 01 14:33:57.523020 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 2158, in _delete
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._local_delete_cleanup(context, instance)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 2117, in _local_delete_cleanup
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._update_queued_for_deletion(context, instance, True)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/compute/api.py", line 2434, in _update_queued_for_deletion
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     im.save()
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     return fn(self, *args, **kwargs)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/opt/stack/new/nova/nova/objects/instance_mapping.py", line 165, in save
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     changes)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1014, in wrapper
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     return fn(*args, **kwargs)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     next(self.gen)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     yield resource
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     next(self.gen)
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 666, in _session
  Jun 01 14:33:57.524852 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self.session.rollback()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self.force_reraise()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     raise value
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 663, in _session
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._end_session_transaction(self.session)
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 691, in _end_session_transaction
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     session.commit()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 1042, in commit
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self.transaction.commit()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 504, in commit
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._prepare_impl()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self.session.flush()
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2523, in flush
  Jun 01 14:33:57.526814 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     self._flush(objects)
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2664, in _flush
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     transaction.rollback(_capture_exception=True)
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     exc_value, with_traceback=exc_tb,
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 178, in raise_
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     raise exception
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2624, in _flush
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     flush_context.execute()
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     rec.execute(self)
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     uow,
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     update,
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 1028, in _emit_update_statements
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi     % (table.description, len(records), rows)
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'instance_mappings' expected to update 1 row(s); 0 were matched.
  Jun 01 14:33:57.528398 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: ERROR nova.api.openstack.wsgi 
  Jun 01 14:33:57.530257 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: INFO nova.api.openstack.wsgi [None req-ab8b5ad1-c168-4f7e-9bfc-42b202b9894f tempest-ServersNegativeTestJSON-1435542876 tempest-ServersNegativeTestJSON-1435542876] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  Jun 01 14:33:57.530257 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: <class 'sqlalchemy.orm.exc.StaleDataError'>
  Jun 01 14:33:57.530257 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: DEBUG nova.api.openstack.wsgi [None req-ab8b5ad1-c168-4f7e-9bfc-42b202b9894f tempest-ServersNegativeTestJSON-1435542876 tempest-ServersNegativeTestJSON-1435542876] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  Jun 01 14:33:57.530257 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: <class 'sqlalchemy.orm.exc.StaleDataError'> {{(pid=13723) __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:941}}
  Jun 01 14:33:57.530257 ubuntu-bionic-rax-iad-0016890725 devstack@n-api.service[13722]: INFO nova.api.openstack.requestlog [None req-ab8b5ad1-c168-4f7e-9bfc-42b202b9894f tempest-ServersNegativeTestJSON-1435542876 tempest-ServersNegativeTestJSON-1435542876] 10.208.194.105 "DELETE /compute/v2.1/servers/d87b9767-d6ac-4c23-ad5b-d1fd139f1662" status: 500 len: 207 microversion: 2.1 time: 0.414127

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1882608/+subscriptions
References

[Bug 1882608] [NEW] DELETE fails with HTTP 500, StaleDataError: UPDATE statement on table 'instance_mappings' expected to update 1 row(s); 0 were matched
From: melanie witt, 2020-06-09