yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1734838] Re: rotate backups failed because of image in use

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1734838@xxxxxxxxxxxxxxxxxx>
Date: Fri, 12 Jan 2018 12:48:33 -0000
Reply-to: Bug 1734838 <1734838@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/523395
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e766e5bd429c6119c077b8be02360ae1e1ec8bd
Submitter: Zuul
Branch:    master

commit 3e766e5bd429c6119c077b8be02360ae1e1ec8bd
Author: Wangpan <wangpan@xxxxxxxxxxxxxxx>
Date:   Tue Nov 28 19:48:21 2017 +0800

    Handle glance exception during rotating instance backup
    
    Glance image backends may return HTTPConflict in many cases
    during deleting image, for example, an rbd image is used for
    booting a new instance, or rbd image has snapshot(s).
    
    If user have already backed up instance to few numbers and then
    execute backup api with rotation 1, then nova will delete the
    previously created images exceeding rotation limit.
    During deleting these images, if the first one of the backup
    images are deleted failed with HTTPConflict or other exceptions,
    all images exceeding rotation limit will be left over.
    
    This patch handles ImageDeleteConflict and all other exceptions
    during deleting backup images, logs a message and continues
    deleting all of the remaining images.
    
    Closes-Bug: #1734838
    
    Change-Id: Ie8091fe3e0e4275717ddc50166345f1c9df4b889


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734838

Title:
  rotate backups failed because of image in use

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  This issue similar to https://bugs.launchpad.net/nova/+bug/1634773,

  Reproduce steps:
  1. create an instance with system disk in rbd backend, named Instance-for-backup
  2. create the first backup of this instance, such as: nova backup Instance-for-backup backup-1 daily 2
  3. create a new instance with this backup-1 snapshot image, such as: nova boot --flavor m1.tiny --image backup-1 ...
  4. then create the second backup of Instance-for-backup, CLI is similar as step 2
  5. finally we create the third backup, we expect the backup-1 image should be rotated out by nova, but it doesn't, and nova-compute report an exception in it's log:

  2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
  7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Rotating out 52 backups _rotate_back
  ups /usr/lib/python2.7/site-packages/nova/compute/manager.py:3262
  2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
  7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Deleting image 9a993e71-71ca-490e-b3
  cb-0b4dca2e574c _rotate_backups /usr/lib/python2.7/site-packages/nova/compute/manager.py:3267
  2017-11-28 16:29:19.280 4154 DEBUG oslo_concurrency.lockutils [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
  bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usag
  e" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
  2017-11-28 16:29:19.353 4154 DEBUG oslo_concurrency.lockutils [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
  bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usag
  e" :: held 0.074s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2017-11-28 16:29:19.354 4154 INFO nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad7
  c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Successfully reverted task state from
   None on failure for instance.
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e
  742bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Exception during message handling: 409 Conflict: Image 9a993e71-71ca-490e-b3
  cb-0b4dca2e574c could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend 
  store outside of Glance. (HTTP 409)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 138, in _dispatch_and_reply
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 185, in _dispatch
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 127, in _do_dispatch
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", li
  ne 110, in wrapped
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     payload)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
  ", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
  ", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3124, in backup_instance
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self._rotate_backups(context, instance, backup_type, rotation)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3269, in _rotate_backups
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.image_api.delete(context, image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 141, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return session.delete(context, image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 499, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self._client.call(context, 1, 'delete', image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 250, in call
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     result = getattr(client.images, method)(*args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 291, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     resp, body = self.client.delete(url)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 287, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return self._request('DELETE', url, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 267, in _request
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     resp, body_iter = self._handle_response(resp)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 83, in _handle_response
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     raise exc.from_response(resp, resp.content)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher HTTPConflict: 409 Conflict: Image 9a993e71-71ca-490e-b3cb-0b4dca2e574c could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance. (HTTP 409)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1734838/+subscriptions
References

[Bug 1734838] [NEW] rotate backups failed because of image in use
From: Wangpan, 2017-11-28