yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2081216] [NEW] openstack server set fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Albert Braden <2081216@xxxxxxxxxxxxxxxxxx>
Date: Thu, 19 Sep 2024 19:56:16 -0000
Reply-to: Bug 2081216 <2081216@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We had a network event and at least one controller lost network access
for a while. After the network was back I created some VMs. One is stuck
in BUILD state. I tried to set it to "ERROR" so that it could be
deleted:

$ os server set --state error 09a04278-70f9-4995-b794-61c362f1fe41
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBReferenceError'> (HTTP 500) (Request-ID: req-8cc82bb9-ddc3-4a85-a87d-ca2183934168)

We're running kolla-ansible Wallaby with libvirt. Our network is Neutron
with openvswitch. Here are the nova-api logs:

2024-09-19 19:41:43.262 50 INFO nova.api.openstack.requestlog [req-2e2699c8-eb85-47ca-9ca3-0858b9b99741 a187a200734f211259c2ca28af7ff440b2616954763d3188d0ca1e5424dce646 076933d285e0424ba846c51e14db006d - b6882f5830c4474d9203b5c5b0ac4ae6 b6882f5830c4474d9203b5c5b0ac4ae6] 10.10.10.12 "GET /v2.1/servers/detail?sort_dir=asc&sort_key=updated_at" status: 200 len: 15 microversion: 2.1 time: 0.059151
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] Unexpected exception in API method: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
[SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, `action`, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
[parameters: {'created_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 729593), 'updated_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'deleted_at': None, 'deleted': 0, 'action': 'resetState', 'instance_uuid': '09a04278-70f9-4995-b794-61c362f1fe41', 'request_id': 'req-8cc82bb9-ddc3-4a85-a87d-ca2183934168', 'user_id': '9ce5d6afb92d440d8819f61cf47c0dc1', 'project_id': '037ee0cfa02645908ce8265b99e08645', 'start_time': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'finish_time': None, 'message': None}]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self.dialect.do_execute(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     cursor.execute(statement, parameters)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = self._query(query)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     conn.query(q)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result.read()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     first_packet = self.connection._read_packet()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     packet.raise_for_error()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi The above exception was the direct cause of the following exception:
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/admin_actions.py", line 71, in _reset_state
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     instance_action = objects.InstanceAction.action_start(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/objects/instance_action.py", line 86, in action_start
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     db_action = db.action_start(context, values)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/api.py", line 1184, in action_start
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return IMPL.action_start(context, values)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 221, in wrapped
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 3769, in action_start
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     action_ref.save(context.session)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/models.py", line 48, in save
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     session.flush()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._flush(objects)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     transaction.rollback(_capture_exception=True)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     compat.raise_(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise exception
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     flush_context.execute()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     rec.execute(self)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     persistence.save_obj(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     _emit_insert_statements(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = cached_connections[connection].execute(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return meth(self, multiparams, params)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return connection._execute_clauseelement(self, multiparams, params)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     ret = self._execute_context(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._handle_dbapi_exception(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise exception
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self.dialect.do_execute(
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     cursor.execute(statement, parameters)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = self._query(query)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     conn.query(q)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result.read()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     first_packet = self.connection._read_packet()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     packet.raise_for_error()
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, `action`, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [parameters: {'created_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 729593), 'updated_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'deleted_at': None, 'deleted': 0, 'action': 'resetState', 'instance_uuid': '09a04278-70f9-4995-b794-61c362f1fe41', 'request_id': 'req-8cc82bb9-ddc3-4a85-a87d-ca2183934168', 'user_id': '9ce5d6afb92d440d8819f61cf47c0dc1', 'project_id': '037ee0cfa02645908ce8265b99e08645', 'start_time': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'finish_time': None, 'message': None}]
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi (Background on this error at: http://sqlalche.me/e/13/gkpj)
2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
2024-09-19 19:41:43.736 26 INFO nova.api.openstack.wsgi [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBReferenceError'>
2024-09-19 19:41:43.737 26 INFO nova.api.openstack.requestlog [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] 10.8.8.41 "POST /v2.1/servers/09a04278-70f9-4995-b794-61c362f1fe41/action" status: 500 len: 208 microversion: 2.1 time: 0.028148

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2081216

Title:
  openstack server set fails

Status in OpenStack Compute (nova):
  New

Bug description:
  We had a network event and at least one controller lost network access
  for a while. After the network was back I created some VMs. One is
  stuck in BUILD state. I tried to set it to "ERROR" so that it could be
  deleted:

  $ os server set --state error 09a04278-70f9-4995-b794-61c362f1fe41
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBReferenceError'> (HTTP 500) (Request-ID: req-8cc82bb9-ddc3-4a85-a87d-ca2183934168)

  We're running kolla-ansible Wallaby with libvirt. Our network is
  Neutron with openvswitch. Here are the nova-api logs:

  2024-09-19 19:41:43.262 50 INFO nova.api.openstack.requestlog [req-2e2699c8-eb85-47ca-9ca3-0858b9b99741 a187a200734f211259c2ca28af7ff440b2616954763d3188d0ca1e5424dce646 076933d285e0424ba846c51e14db006d - b6882f5830c4474d9203b5c5b0ac4ae6 b6882f5830c4474d9203b5c5b0ac4ae6] 10.10.10.12 "GET /v2.1/servers/detail?sort_dir=asc&sort_key=updated_at" status: 200 len: 15 microversion: 2.1 time: 0.059151
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] Unexpected exception in API method: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
  [SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, `action`, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
  [parameters: {'created_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 729593), 'updated_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'deleted_at': None, 'deleted': 0, 'action': 'resetState', 'instance_uuid': '09a04278-70f9-4995-b794-61c362f1fe41', 'request_id': 'req-8cc82bb9-ddc3-4a85-a87d-ca2183934168', 'user_id': '9ce5d6afb92d440d8819f61cf47c0dc1', 'project_id': '037ee0cfa02645908ce8265b99e08645', 'start_time': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'finish_time': None, 'message': None}]
  (Background on this error at: http://sqlalche.me/e/13/gkpj)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self.dialect.do_execute(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     cursor.execute(statement, parameters)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = self._query(query)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     conn.query(q)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result.read()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     first_packet = self.connection._read_packet()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     packet.raise_for_error()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi The above exception was the direct cause of the following exception:
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/admin_actions.py", line 71, in _reset_state
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     instance_action = objects.InstanceAction.action_start(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/objects/instance_action.py", line 86, in action_start
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     db_action = db.action_start(context, values)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/api.py", line 1184, in action_start
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return IMPL.action_start(context, values)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 221, in wrapped
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 3769, in action_start
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     action_ref.save(context.session)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/models.py", line 48, in save
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     session.flush()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._flush(objects)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     transaction.rollback(_capture_exception=True)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     compat.raise_(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise exception
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     flush_context.execute()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     rec.execute(self)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     persistence.save_obj(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     _emit_insert_statements(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = cached_connections[connection].execute(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return meth(self, multiparams, params)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     return connection._execute_clauseelement(self, multiparams, params)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     ret = self._execute_context(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._handle_dbapi_exception(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise exception
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self.dialect.do_execute(
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     cursor.execute(statement, parameters)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result = self._query(query)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     conn.query(q)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     result.read()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     first_packet = self.connection._read_packet()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     packet.raise_for_error()
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, `action`, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi [parameters: {'created_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 729593), 'updated_at': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'deleted_at': None, 'deleted': 0, 'action': 'resetState', 'instance_uuid': '09a04278-70f9-4995-b794-61c362f1fe41', 'request_id': 'req-8cc82bb9-ddc3-4a85-a87d-ca2183934168', 'user_id': '9ce5d6afb92d440d8819f61cf47c0dc1', 'project_id': '037ee0cfa02645908ce8265b99e08645', 'start_time': datetime.datetime(2024, 9, 19, 19, 41, 43, 710866), 'finish_time': None, 'message': None}]
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi (Background on this error at: http://sqlalche.me/e/13/gkpj)
  2024-09-19 19:41:43.732 26 ERROR nova.api.openstack.wsgi
  2024-09-19 19:41:43.736 26 INFO nova.api.openstack.wsgi [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBReferenceError'>
  2024-09-19 19:41:43.737 26 INFO nova.api.openstack.requestlog [req-8cc82bb9-ddc3-4a85-a87d-ca2183934168 9ce5d6afb92d440d8819f61cf47c0dc1 037ee0cfa02645908ce8265b99e08645 - default default] 10.8.8.41 "POST /v2.1/servers/09a04278-70f9-4995-b794-61c362f1fe41/action" status: 500 len: 208 microversion: 2.1 time: 0.028148

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2081216/+subscriptions