← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #12286

[Bug 1298216] [NEW] cold migrate with libvirt driver failed because of ImageNotFound exception

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The reason is that, nova wants to cache the base image even if it is
doing a cold migration operation, this is a redundant and wrong thing,
when the image is deleted, then an ImageNotFound exception is raised.

reproduce steps:
1) create an instance
2) delete the image of instance
3) cold migrate the instance to another host(be sure the image cache is not exists on this host)

then you will find your instance becomes to ERROR status, and trace log in compute log:
2014-03-27 13:46:20.346 25728 ERROR nova.openstack.common.rpc.amqp [req-8bad3379-cdb9-4e89-b4f7-feeaf0b453f5 5cf8d1ba0d6042268451e1c1c82f0a00 6d57301878904b1db2ee4d93a1d1b028] Exception during message handling
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp **args)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 414, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp payload)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 304, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp pass
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 332, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3645, in finish_resize
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3633, in finish_resize
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp disk_info, image)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3596, in _finish_resize
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp block_device_info, power_on)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5402, in finish_migration
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp block_device_info=None, inject_files=False)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2990, in _create_image
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp project_id=instance['project_id'])
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 174, in cache
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 276, in create_image
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp prepare_template(target=base, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 248, in inner
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 162, in call_if_not_exists
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp fetch_func(target=target, *args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 693, in fetch_image
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp images.fetch_to_raw(context, image_id, target, user_id, project_id)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 195, in fetch_to_raw
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp fetch(context, image_href, path_tmp, user_id, project_id)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp image_service.download(context, image_id, dst_path=path)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 349, in download
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp _reraise_translated_image_exception(image_id)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 347, in download
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp image_chunks = self._client.call(context, 1, 'data', image_id)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 212, in call
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return getattr(client.images, method)(*args, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 127, in data
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp % urllib.quote(str(image_id)))
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in raw_request
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return self._http_request(url, method, **kwargs)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 233, in _http_request
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp raise exc.from_response(resp, body_str)
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp ImageNotFound: Image 9fe5f3d9-8d00-4962-bc0f-afe7eb8438b3 could not be found.
2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1298216

Title:
  cold migrate with libvirt driver failed because of ImageNotFound
  exception

Status in OpenStack Compute (Nova):
  New

Bug description:
  The reason is that, nova wants to cache the base image even if it is
  doing a cold migration operation, this is a redundant and wrong thing,
  when the image is deleted, then an ImageNotFound exception is raised.

  reproduce steps:
  1) create an instance
  2) delete the image of instance
  3) cold migrate the instance to another host(be sure the image cache is not exists on this host)

  then you will find your instance becomes to ERROR status, and trace log in compute log:
  2014-03-27 13:46:20.346 25728 ERROR nova.openstack.common.rpc.amqp [req-8bad3379-cdb9-4e89-b4f7-feeaf0b453f5 5cf8d1ba0d6042268451e1c1c82f0a00 6d57301878904b1db2ee4d93a1d1b028] Exception during message handling
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp **args)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 414, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp payload)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 304, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp pass
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 332, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3645, in finish_resize
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3633, in finish_resize
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp disk_info, image)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3596, in _finish_resize
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp block_device_info, power_on)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5402, in finish_migration
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp block_device_info=None, inject_files=False)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2990, in _create_image
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp project_id=instance['project_id'])
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 174, in cache
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 276, in create_image
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp prepare_template(target=base, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 248, in inner
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 162, in call_if_not_exists
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp fetch_func(target=target, *args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 693, in fetch_image
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp images.fetch_to_raw(context, image_id, target, user_id, project_id)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 195, in fetch_to_raw
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp fetch(context, image_href, path_tmp, user_id, project_id)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp image_service.download(context, image_id, dst_path=path)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 349, in download
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp _reraise_translated_image_exception(image_id)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 347, in download
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp image_chunks = self._client.call(context, 1, 'data', image_id)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 212, in call
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return getattr(client.images, method)(*args, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 127, in data
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp % urllib.quote(str(image_id)))
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in raw_request
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp return self._http_request(url, method, **kwargs)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 233, in _http_request
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp raise exc.from_response(resp, body_str)
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp ImageNotFound: Image 9fe5f3d9-8d00-4962-bc0f-afe7eb8438b3 could not be found.
  2014-03-27 13:46:20.346 25728 TRACE nova.openstack.common.rpc.amqp

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1298216/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next