yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1860990] Re: RBD image backend tries to flatten images even if they are already flat

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Elod Illes <1860990@xxxxxxxxxxxxxxxxxx>
Date: Tue, 24 Nov 2020 14:08:50 -0000
Reply-to: Bug 1860990 <1860990@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/train
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860990

Title:
  RBD image backend tries to flatten images even if they are already
  flat

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  Fix Committed
Status in OpenStack Compute (nova) rocky series:
  Fix Committed
Status in OpenStack Compute (nova) stein series:
  Fix Released
Status in OpenStack Compute (nova) train series:
  Fix Released

Bug description:
  When [DEFAULT]show_multiple_locations option is not set in glance, and
  both glance and nova use ceph as their backend, with properly
  configured accesses, nova will fail with the following exception:

  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [req-8021fd76-d5ab-4a9b-bd17-f5eb4d4faf62 0e96a04f360644818632b7e46fe8d3e7 ac01daacc7424a40b8b464a163902dcb - default default] [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Instance failed to spawn: rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Traceback (most recent call last):
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 5757, in _unshelve_instance
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     block_device_info=block_device_info)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3457, in spawn
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     block_device_info=block_device_info)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3832, in _create_image
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     fallback_from_host)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3923, in _create_and_inject_local_root
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     instance, size, fallback_from_host)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 9267, in _try_fetch_image_cache
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     image.flatten()
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/imagebackend.py", line 983, in flatten
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     self.driver.flatten(self.rbd_name, pool=self.driver.pool)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 290, in flatten
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     vol.flatten()
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     rv = execute(f, *args, **kwargs)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     six.reraise(c, e, tb)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     raise value
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]     rv = meth(*args, **kwargs)
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]   File "rbd.pyx", line 2207, in rbd.Image.flatten
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
  2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]

  During unshelve, when nova fails to determine the parent pool for the
  image (because glance api does not return rbd image location), it
  downloads the image through the glance api and puts it into its pool.
  Such image will be already flat. And nova will try to flatten it again
  and fail. It might be better to make flatten idempotent and just be a
  noop for already flat images.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860990/+subscriptions
References

[Bug 1860990] [NEW] RBD image backend tries to flatten images even if they are already flat
From: Vladyslav Drok, 2020-01-27