yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88610
[Bug 1915400] Re: Snapshots fail with traceback from API
Reviewed: https://review.opendev.org/c/openstack/nova/+/830383
Committed: https://opendev.org/openstack/nova/commit/b2d28f890872747d099a262e4a208e146b882f3f
Submitter: "Zuul (22348)"
Branch: master
commit b2d28f890872747d099a262e4a208e146b882f3f
Author: Balazs Gibizer <balazs.gibizer@xxxxxxxx>
Date: Tue Feb 22 12:24:37 2022 +0100
Fix eventlet.tpool import
Currently nova.utils.tpool_execute() only works by chance. And as the
bug report shows there are env where it fails.
The nova.utils.tpool_execute() call tries to uses eventlet.tpool.execute
but the tpool module is not imported by the utils module only eventlet.
In devstack it works by chance as the wsgi init actually imports
eventlet.tpool indirectly via:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/stack/nova/nova/api/openstack/compute/__init__.py", line 21, in <module>
from nova.api.openstack.compute.routes import APIRouterV21 # noqa
File "/opt/stack/nova/nova/api/openstack/compute/routes.py", line 20, in <module>
from nova.api.openstack.compute import admin_actions
File "/opt/stack/nova/nova/api/openstack/compute/admin_actions.py", line 17, in <module>
from nova.api.openstack import common
File "/opt/stack/nova/nova/api/openstack/common.py", line 27, in <module>
from nova.compute import task_states
File "/opt/stack/nova/nova/compute/task_states.py", line 26, in <module>
from nova.objects import fields
File "/opt/stack/nova/nova/objects/fields.py", line 24, in <module>
from nova.network import model as network_model
File "/opt/stack/nova/nova/network/model.py", line 23, in <module>
from nova import utils
File "/opt/stack/nova/nova/utils.py", line 39, in <module>
from oslo_concurrency import processutils
File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 57, in <module>
from eventlet import tpool
This was broken since I8dbc579e0037969aab4f2bb500fccfbde4190726. This
patch adds the correct import statement.
Change-Id: Ic46345ceeb445164aea6ae9b35c457c6150765f6
Closes-Bug: #1915400
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915400
Title:
Snapshots fail with traceback from API
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) victoria series:
New
Status in OpenStack Compute (nova) wallaby series:
New
Status in OpenStack Compute (nova) xena series:
New
Bug description:
Description
===========
Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as:
Error: Unable to create snapshot. Details
Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible. <class 'AttributeError'> (HTTP 500) (Request-ID:
req-7e4b0824-9884-4d50-b8ee-5184e245a11d)
It appears that this can be worked around by adding 'from eventlet
import tpool' and adjusting the 'eventlet.tpool.execute' call to match
in nova/utils.py
Steps to reproduce
==================
Via Horizon on a Victoria deployment:
- Create an instance
- Suspend the instance
- Create a snapshot
Expected result
===============
A snapshot should have been created
Actual result
=============
Error message as above
Environment
===========
1. Exact version of OpenStack you are running.
Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6
2. Which hypervisor did you use?
Libvirt+KVM
2. Which storage type did you use?
Ceph Octopus
3. Which networking type did you use?
Neutron with Linux Bridge
Logs & Configs
==============
The full traceback from the API is as follows:
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data))
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'AttributeError'>
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915400/+subscriptions
References
