yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85136
[Bug 1915400] [NEW] Snapshots fail with traceback from API
Public bug reported:
Description
===========
Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as:
Error: Unable to create snapshot. Details
Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'AttributeError'> (HTTP 500) (Request-ID: req-7e4b0824-9884-4d50
-b8ee-5184e245a11d)
It appears that this can be worked around by adding 'from eventlet
import tpool' and adjusting the 'eventlet.tpool.execute' call to match
in nova/utils.py
Steps to reproduce
==================
Via Horizon on a Victoria deployment:
- Create an instance
- Suspend the instance
- Create a snapshot
Expected result
===============
A snapshot should have been created
Actual result
=============
Error message as above
Environment
===========
1. Exact version of OpenStack you are running.
Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6
2. Which hypervisor did you use?
Libvirt+KVM
2. Which storage type did you use?
Ceph Octopus
3. Which networking type did you use?
Neutron with Linux Bridge
Logs & Configs
==============
The full traceback from the API is as follows:
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data))
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'AttributeError'>
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1915400
Title:
Snapshots fail with traceback from API
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Having upgraded three OpenStack deployments to Victoria, we have noticed that snapshots are now failing to be created. When a user attempts this via Horizon they receive an error such as:
Error: Unable to create snapshot. Details
Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible. <class 'AttributeError'> (HTTP 500) (Request-ID: req-
7e4b0824-9884-4d50-b8ee-5184e245a11d)
It appears that this can be worked around by adding 'from eventlet
import tpool' and adjusting the 'eventlet.tpool.execute' call to match
in nova/utils.py
Steps to reproduce
==================
Via Horizon on a Victoria deployment:
- Create an instance
- Suspend the instance
- Create a snapshot
Expected result
===============
A snapshot should have been created
Actual result
=============
Error message as above
Environment
===========
1. Exact version of OpenStack you are running.
Git hash cb4963ba8c4babb7e460d71daf91099fc6da1dc6
2. Which hypervisor did you use?
Libvirt+KVM
2. Which storage type did you use?
Ceph Octopus
3. Which networking type did you use?
Neutron with Linux Bridge
Logs & Configs
==============
The full traceback from the API is as follows:
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] Unexpected exception in API method: AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1244, in _action_create_image
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi metadata)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 156, in inner
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/compute/api.py", line 3374, in snapshot_volume_backed
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 1224, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 555, in create
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi sharing_member_id=sharing_member_id)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 670, in _create_v2
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi image = self._upload_data(context, image_id, data)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/image/glance.py", line 591, in _upload_data
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi args=(image_id, data))
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi File "/openstack/venvs/nova-22.0.0/lib/python3.6/site-packages/nova/utils.py", line 694, in tpool_execute
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi eventlet.tpool.execute(func, *args, **kwargs)
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi AttributeError: module 'eventlet' has no attribute 'tpool'
2021-02-11 11:11:48.069 26063 ERROR nova.api.openstack.wsgi
Feb 11 11:11:48 infra3-nova-api-container-75132bde nova-api-wsgi[26063]: 2021-02-11 11:11:48.079 26063 INFO nova.api.openstack.wsgi [req-cc475f2d-9575-407e-b88f-41545e13964b 6c10bb801323c78ca3b191e70263ed3eb195bfdfe4d98ab964bf03514271a7d3 2d449695e7cb4e5aa5abdc8cccbaa36b - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'AttributeError'>
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1915400/+subscriptions
Follow ups
