yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1736759] [NEW] Glance images can contain no data

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Stephen Finucane <sfinucan@xxxxxxxxxx>
Date: Wed, 06 Dec 2017 16:52:57 -0000
Reply-to: Bug 1736759 <1736759@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Due to another bug [1], glance was returning None from
'glanceclient.v2.images.Controller.data'. However, the glance
documentation states that this is a valid return value. We should handle
this. Logs below.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1476448
[2] https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data

---

2017-08-15 17:34:01.677 1 ERROR nova.image.glance [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Error writing to /var/lib/nova/instances/_base/cae3a4306eeb5643cb6caffbe1e3050645f8aee2.part: 'NoneType' object is not iterable
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Instance failed to spawn: TypeError: 'NoneType' object is not iterable
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Traceback (most recent call last):
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     yield resources
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     block_device_info=block_device_info)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     block_device_info=block_device_info)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fallback_from_host)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     instance, size, fallback_from_host)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     size=size)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     *args, **kwargs)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 595, in create_image
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     prepare_template(target=base, *args, **kwargs)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     return f(*args, **kwargs)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 237, in fetch_func_sync
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fetch_func(target=target, *args, **kwargs)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 446, in fetch_image
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     images.fetch_to_raw(context, image_id, target)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 134, in fetch_to_raw
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fetch(context, image_href, path_tmp)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 125, in fetch
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     IMAGE_API.download(context, image_href, dest_path=path)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 184, in download
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     dst_path=dest_path)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 369, in download
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     {'path': dst_path, 'exception': ex})
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     self.force_reraise()
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     six.reraise(self.type_, self.value, self.tb)
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 353, in download
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     for chunk in image_chunks:
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable
2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]

** Affects: nova
     Importance: Undecided
     Assignee: Stephen Finucane (stephenfinucane)
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1736759

Title:
  Glance images can contain no data

Status in OpenStack Compute (nova):
  New

Bug description:
  Due to another bug [1], glance was returning None from
  'glanceclient.v2.images.Controller.data'. However, the glance
  documentation states that this is a valid return value. We should
  handle this. Logs below.

  [1] https://bugzilla.redhat.com/show_bug.cgi?id=1476448
  [2] https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data

  ---

  2017-08-15 17:34:01.677 1 ERROR nova.image.glance [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] Error writing to /var/lib/nova/instances/_base/cae3a4306eeb5643cb6caffbe1e3050645f8aee2.part: 'NoneType' object is not iterable
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [req-70546b57-a282-4552-8b9e-65be1871825a bd800a91d263411393899aff269084a0 aaed41f2e25f494c9fadd01c340f25c8 - default default] [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Instance failed to spawn: TypeError: 'NoneType' object is not iterable
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] Traceback (most recent call last):
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     yield resources
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1940, in _build_and_run_instance
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     block_device_info=block_device_info)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2793, in spawn
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     block_device_info=block_device_info)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3231, in _create_image
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fallback_from_host)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3322, in _create_and_inject_local_root
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     instance, size, fallback_from_host)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6968, in _try_fetch_image_cache
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     size=size)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in cache
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     *args, **kwargs)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 595, in create_image
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     prepare_template(target=base, *args, **kwargs)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     return f(*args, **kwargs)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 237, in fetch_func_sync
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fetch_func(target=target, *args, **kwargs)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 446, in fetch_image
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     images.fetch_to_raw(context, image_id, target)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 134, in fetch_to_raw
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     fetch(context, image_href, path_tmp)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 125, in fetch
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     IMAGE_API.download(context, image_href, dest_path=path)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 184, in download
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     dst_path=dest_path)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 369, in download
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     {'path': dst_path, 'exception': ex})
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     self.force_reraise()
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     six.reraise(self.type_, self.value, self.tb)
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 353, in download
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]     for chunk in image_chunks:
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0] TypeError: 'NoneType' object is not iterable
  2017-08-15 17:34:01.679 1 ERROR nova.compute.manager [instance: c3fc31f1-28ab-47bf-a08b-65cfc6ab2ce0]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1736759/+subscriptions
Follow ups

[Bug 1736759] Re: Glance images can contain no data
From: OpenStack Infra, 2018-01-27
[Bug 1736759] Re: Glance images can contain no data
From: OpenStack Infra, 2018-01-02
[Bug 1736759] Re: Glance images can contain no data
From: Stephen Finucane, 2017-12-08