yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1734838] Re: rotate backups failed because of image in use

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 08 Mar 2018 14:03:10 -0000
Reply-to: Bug 1734838 <1734838@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Changed in: nova/pike
       Status: New => In Progress

** Changed in: nova/pike
   Importance: Undecided => Low

** Changed in: nova/pike
     Assignee: (unassigned) => Wangpan (aspirer)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1734838

Title:
  rotate backups failed because of image in use

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  In Progress

Bug description:
  This issue similar to https://bugs.launchpad.net/nova/+bug/1634773,

  Reproduce steps:
  1. create an instance with system disk in rbd backend, named Instance-for-backup
  2. create the first backup of this instance, such as: nova backup Instance-for-backup backup-1 daily 2
  3. create a new instance with this backup-1 snapshot image, such as: nova boot --flavor m1.tiny --image backup-1 ...
  4. then create the second backup of Instance-for-backup, CLI is similar as step 2
  5. finally we create the third backup, we expect the backup-1 image should be rotated out by nova, but it doesn't, and nova-compute report an exception in it's log:

  2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
  7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Rotating out 52 backups _rotate_back
  ups /usr/lib/python2.7/site-packages/nova/compute/manager.py:3262
  2017-11-28 16:29:16.361 4154 DEBUG nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad
  7c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Deleting image 9a993e71-71ca-490e-b3
  cb-0b4dca2e574c _rotate_backups /usr/lib/python2.7/site-packages/nova/compute/manager.py:3267
  2017-11-28 16:29:19.280 4154 DEBUG oslo_concurrency.lockutils [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
  bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usag
  e" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
  2017-11-28 16:29:19.353 4154 DEBUG oslo_concurrency.lockutils [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742
  bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usag
  e" :: held 0.074s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
  2017-11-28 16:29:19.354 4154 INFO nova.compute.manager [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e742bab1ad7
  c 0e5bcf89990c455882649ed88b32e27d - - -] [instance: 00bce146-5408-4c33-bc11-7458f847eb19] Successfully reverted task state from
   None on failure for instance.
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher [req-56a39c62-2010-4004-aaac-9c3e2669ee4d af175a5930e2470c8725e
  742bab1ad7c 0e5bcf89990c455882649ed88b32e27d - - -] Exception during message handling: 409 Conflict: Image 9a993e71-71ca-490e-b3
  cb-0b4dca2e574c could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend 
  store outside of Glance. (HTTP 409)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 138, in _dispatch_and_reply
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 185, in _dispatch
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dis
  patcher.py", line 127, in _do_dispatch
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", li
  ne 110, in wrapped
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     payload)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
  ", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py
  ", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3124, in backup_instance
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self._rotate_backups(context, instance, backup_type, rotation)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3269, in _rotate_backups
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self.image_api.delete(context, image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 141, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return session.delete(context, image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 499, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     self._client.call(context, 1, 'delete', image_id)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 250, in call
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     result = getattr(client.images, method)(*args, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 291, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     resp, body = self.client.delete(url)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 287, in delete
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     return self._request('DELETE', url, **kwargs)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 267, in _request
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     resp, body_iter = self._handle_response(resp)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 83, in _handle_response
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher     raise exc.from_response(resp, resp.content)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher HTTPConflict: 409 Conflict: Image 9a993e71-71ca-490e-b3cb-0b4dca2e574c could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance. (HTTP 409)
  2017-11-28 16:29:19.361 4154 ERROR oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1734838/+subscriptions
References

[Bug 1734838] [NEW] rotate backups failed because of image in use
From: Wangpan, 2017-11-28