yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1477878] [NEW] Unable to launch instances from snapshot due to kernel and ramdisk fields in glance database

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vj <acetone_black@xxxxxxxxxxx>
Date: Fri, 24 Jul 2015 08:08:48 -0000
Reply-to: Bug 1477878 <1477878@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi All,

I am using openstack kilo with ceph backend. Creating a snapshot of an
instance works fine, but launching an instance from the snapshot fails.
Corresponding nova logs:

-----------------------------------------------
2015-07-24 12:46:44.918 7176 ERROR nova.compute.manager [req-f2bfa4ae-20d4-4f10-8772-ab8b1993260a - - - - -] [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Instance failed to spawn
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Traceback (most recent call last):
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     yield resources
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     block_device_info=block_device_info)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2347, in spawn
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     admin_pass=admin_password)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2745, in _create_image
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     instance, size, fallback_from_host)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5875, in _try_fetch_image_cache
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     size=size)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     *args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 727, in create_image
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     prepare_template(target=base, max_size=size, *args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return f(*args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     fetch_func(target=target, *args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2737, in clone_fallback_to_fetch
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     backend.clone(context, disk_images['image_id'])
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 752, in clone
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     include_locations=True)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 93, in get
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     show_deleted=show_deleted)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 301, in show
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     _reraise_translated_image_exception(image_id)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 299, in show
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     image = self._client.call(context, version, 'get', image_id)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 219, in call
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return getattr(client.images, method)(*args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 106, in get
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return self.model(**body)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/warlock/core.py", line 32, in __init__
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     base_class.__init__(self, *args, **kwargs)
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/warlock/model.py", line 35, in __init__
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     raise ValueError(str(exc))
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] ValueError: None is not of type u'string'
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Failed validating u'type' in schema[u'properties'][u'ramdisk_id']:
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     {u'description': u'ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.',
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'is_base': False,
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'pattern': u'^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$',
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'type': u'string'}
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] On instance[u'ramdisk_id']:
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     None
2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
-----------------------------------------------

I have found similar bugs: https://bugs.launchpad.net/python-
glanceclient/+bug/1447193, but I am able to list images with glance
after snapshot.

This is due to the presence of ramdisk_id and kernel_id fields in
image_properties table of glance database. When I delete these 2 fields,
then I am able to boot instances from the snapshot successfully. I have
attached the glance and nova config files.

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova-glance-configs.tar.gz"
   https://bugs.launchpad.net/bugs/1477878/+attachment/4433468/+files/nova-glance-configs.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1477878

Title:
  Unable to launch instances from snapshot due to kernel and ramdisk
  fields in glance database

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi All,

  I am using openstack kilo with ceph backend. Creating a snapshot of an
  instance works fine, but launching an instance from the snapshot
  fails. Corresponding nova logs:

  -----------------------------------------------
  2015-07-24 12:46:44.918 7176 ERROR nova.compute.manager [req-f2bfa4ae-20d4-4f10-8772-ab8b1993260a - - - - -] [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Instance failed to spawn
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Traceback (most recent call last):
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     yield resources
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     block_device_info=block_device_info)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2347, in spawn
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     admin_pass=admin_password)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2745, in _create_image
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     instance, size, fallback_from_host)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5875, in _try_fetch_image_cache
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     size=size)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     *args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 727, in create_image
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     prepare_template(target=base, max_size=size, *args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return f(*args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     fetch_func(target=target, *args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2737, in clone_fallback_to_fetch
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     backend.clone(context, disk_images['image_id'])
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 752, in clone
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     include_locations=True)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 93, in get
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     show_deleted=show_deleted)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 301, in show
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     _reraise_translated_image_exception(image_id)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 299, in show
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     image = self._client.call(context, version, 'get', image_id)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 219, in call
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return getattr(client.images, method)(*args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 106, in get
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     return self.model(**body)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/warlock/core.py", line 32, in __init__
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     base_class.__init__(self, *args, **kwargs)
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]   File "/usr/lib/python2.7/dist-packages/warlock/model.py", line 35, in __init__
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     raise ValueError(str(exc))
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] ValueError: None is not of type u'string'
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] Failed validating u'type' in schema[u'properties'][u'ramdisk_id']:
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     {u'description': u'ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.',
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'is_base': False,
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'pattern': u'^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$',
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]      u'type': u'string'}
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] On instance[u'ramdisk_id']:
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc]     None
  2015-07-24 12:46:44.918 7176 TRACE nova.compute.manager [instance: b05daf8c-818f-4018-8790-8f03d44d2fcc] 
  -----------------------------------------------

  I have found similar bugs: https://bugs.launchpad.net/python-
  glanceclient/+bug/1447193, but I am able to list images with glance
  after snapshot.

  This is due to the presence of ramdisk_id and kernel_id fields in
  image_properties table of glance database. When I delete these 2
  fields, then I am able to boot instances from the snapshot
  successfully. I have attached the glance and nova config files.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1477878/+subscriptions
Follow ups

[Bug 1477878] Re: Unable to launch instances from snapshot due to kernel and ramdisk fields in glance database
From: Vj, 2015-08-11