yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1940812] Re: UnboundLocalError: local variable 'instance_uuid' referenced before assignment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1940812@xxxxxxxxxxxxxxxxxx>
Date: Wed, 03 Nov 2021 19:51:41 -0000
Reply-to: Bug 1940812 <1940812@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/nova/+/805605
Committed: https://opendev.org/openstack/nova/commit/14e43f385e6d243b6efd11a777d082e63b66367c
Submitter: "Zuul (22348)"
Branch:    master

commit 14e43f385e6d243b6efd11a777d082e63b66367c
Author: Balazs Gibizer <balazs.gibizer@xxxxxxxx>
Date:   Mon Aug 23 10:56:58 2021 +0200

    Avoid unbound instance_uuid var during delete
    
    The patch I03cf285ad83e09d88cdb702a88dfed53c01610f8 fixed most of the
    possible cases for this to happen but missed one. An early enough
    exception during _delete() can cause that the instance_uuid never gets
    defined but then we try to use it during the finally block. This patch
    moves the saving of the instance_uuid to the top of the try block to
    avoid the issue.
    
    Change-Id: Ib3073d7f595c8927532b7c49fc7e5ffe80d508b9
    Closes-Bug: #1940812
    Related-Bug: #1914777


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1940812

Title:
  UnboundLocalError: local variable 'instance_uuid' referenced before
  assignment

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  there is an unbound variable in [1] causing:

  
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi [None req-b45811ea-c6f9-4422-8cf1-8970273587bc tempest-DeleteServersTestJSON-1241246976 tempest-DeleteServersTestJSON-1241246976-project] Unexpected exception in API method: UnboundLocalError: local variable 'instance_uuid' referenced before assignment
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/objects/instance.py", line 653, in destroy
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     db_inst = db.instance_destroy(self._context, self.uuid,
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     ectxt.value = e.inner_exc
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self.force_reraise()
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     raise self.value
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/db/main/api.py", line 1294, in instance_destroy
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     raise exception.ConstraintNotMet()
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi nova.exception.ConstraintNotMet: Constraint not met.
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi 
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi 
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2248, in _delete
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     if self._delete_while_booting(context, instance):
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2188, in _delete_while_booting
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     instance.destroy()
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return fn(self, *args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/objects/instance.py", line 658, in destroy
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     raise exception.ObjectActionError(action='destroy',
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi nova.exception.ObjectActionError: Object action destroy failed because: host changed
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi 
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi 
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 658, in wrapped
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1072, in delete
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self._delete(req.environ['nova.context'], req, id)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 884, in _delete
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self.compute_api.delete(context, instance)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 228, in inner
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 155, in inner
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2575, in delete
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self._delete_instance(context, instance)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2566, in _delete_instance
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self._delete(context, instance, 'delete', self._do_delete,
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi   File "/opt/stack/nova/nova/compute/api.py", line 2290, in _delete
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi     self._local_delete_cleanup(context, instance_uuid)
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi UnboundLocalError: local variable 'instance_uuid' referenced before assignment
  Aug 20 14:57:42.704720 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: ERROR nova.api.openstack.wsgi 
  Aug 20 14:57:42.707787 ubuntu-focal-inap-mtl01-0026012484 devstack@n-api.service[111846]: INFO nova.api.openstack.wsgi [None req-b45811ea-c6f9-4422-8cf1-8970273587bc tempest-DeleteServersTestJSON-1241246976 tempest-DeleteServersTestJSON-1241246976-project] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

  
  [1] https://github.com/openstack/nova/blob/10b1dc84f47a71061340f8e0ae0fe32dca44061a/nova/compute/api.py#L2290
  [2] https://61871398d16bb191f50f-b8e5e06ecccd85e574a730a33d1fddb0.ssl.cf2.rackcdn.com/745605/6/gate/nova-ceph-multistore/805eaaa/controller/logs/screen-n-api.txt

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1940812/+subscriptions
References

[Bug 1940812] [NEW] UnboundLocalError: local variable 'instance_uuid' referenced before assignment
From: Balazs Gibizer, 2021-08-23