yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1634773] [NEW] backup instance won't be able to delete all images exceeding rotation limit, if user mistakenly deletes one of the backup image

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bhagyashri Shewale <bhagyashri.shewale@xxxxxxxxxxx>
Date: Wed, 19 Oct 2016 07:35:26 -0000
Reply-to: Bug 1634773 <1634773@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If user have already backed up instance to few numbers and then execute
backup api with rotation 1, then nova will delete the previously
created images exceeding rotation limit.

During deleting these images, if user mistakenly deletes one of the
image in advance, then backup instance won't be able to delete all
images exceeding rotation limit causing api failure.

Steps to reproduce:

 1. Create instance
 
  $ nova boot --flavor <flavor_id> --image <image_id> <server_name>
 
  $ nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec test
  
  +--------------------------------------+------+--------+------------+-------------+---------------------------------+
  | ID                                   | Name | Status | Task State | Power State | Networks                        |
  +--------------------------------------+------+--------+------------+-------------+---------------------------------+
  | a9e41bc5-516c-48f1-bed8-2475650835c8 | test | ACTIVE | -          | Running     | public=2001:db8::8, 172.24.4.12 |
  +--------------------------------------+------+--------+------------+-------------+---------------------------------+

  
 2. Create 5 instance backup using the backup api(hit below command 5 times)
 
  $ nova backup <server> <name> <backup-type> <rotation>
 
  $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 5

 3. Take list of backup images
 
  $ glance image-list
  
  +--------------------------------------+---------------------------------+
  | ID                                   | Name                            |
  +--------------------------------------+---------------------------------+
  | c33790be-df8b-4740-8311-0fa787bfb087 | cirros-0.3.4-x86_64-uec         |
  | de43a375-6d6d-4a30-856e-7a68db083834 | cirros-0.3.4-x86_64-uec-kernel  |
  | 0a6c800e-6002-4f2f-a2b9-0542e0f2fac0 | cirros-0.3.4-x86_64-uec-ramdisk |
  | fc0d2030-1cec-4780-9199-e99c63da0005 | test_bk                         |
  | a9d7dc78-acab-4e4e-b03d-c7da9cb5ef00 | test_bk                         |
  | 937061b4-51c7-4835-b9b7-d63464665a27 | test_bk                         |
  | 41e25c26-c61e-421c-b170-a39539bcbc78 | test_bk                         |
  | a15ae46a-a3c3-49a8-be75-1aee724fa4e9 | test_bk                         |
  +--------------------------------------+---------------------------------+

  
 4. Add debug point using pdb.set_trace() in nova.compute.manager._rotate_backups method [1]
  
 5. Delete one of the backup images from above list
 
  $ glance image-delete fc0d2030-1cec-4780-9199-e99c63da0005

 6. Create the insatnce backup by passing the rotation parameter value as 1.
  
  $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 1
 
 7. Checked the n-cpu logs
 
2016-10-17 15:59:38.978 ERROR oslo_messaging.rpc.server [req-99fcf837-514a-4a80-a19e-c8ac2f26768f admin admin] Exception during message handling
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 179, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     instance=instance)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3001, in backup_instance
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self._rotate_backups(context, instance, backup_type, rotation)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3145, in _rotate_backups
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.image_api.delete(context, image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/api.py", line 141, in delete
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return session.delete(context, image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/glance.py", line 765, in delete
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     raise exception.ImageNotFound(image_id=image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server ImageNotFound: Image fc0d2030-1cec-4780-9199-e99c63da0005 could not be found.
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server
 

[1]
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3139

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1634773

Title:
  backup instance won't be able to delete all images exceeding rotation
  limit, if user mistakenly deletes one of the backup image

Status in OpenStack Compute (nova):
  New

Bug description:
  If user have already backed up instance to few numbers and then execute
  backup api with rotation 1, then nova will delete the previously
  created images exceeding rotation limit.

  During deleting these images, if user mistakenly deletes one of the
  image in advance, then backup instance won't be able to delete all
  images exceeding rotation limit causing api failure.

  Steps to reproduce:

   1. Create instance
   
    $ nova boot --flavor <flavor_id> --image <image_id> <server_name>
   
    $ nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec test
    
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+
    | ID                                   | Name | Status | Task State | Power State | Networks                        |
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+
    | a9e41bc5-516c-48f1-bed8-2475650835c8 | test | ACTIVE | -          | Running     | public=2001:db8::8, 172.24.4.12 |
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+

    
   2. Create 5 instance backup using the backup api(hit below command 5 times)
   
    $ nova backup <server> <name> <backup-type> <rotation>
   
    $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 5

   3. Take list of backup images
   
    $ glance image-list
    
    +--------------------------------------+---------------------------------+
    | ID                                   | Name                            |
    +--------------------------------------+---------------------------------+
    | c33790be-df8b-4740-8311-0fa787bfb087 | cirros-0.3.4-x86_64-uec         |
    | de43a375-6d6d-4a30-856e-7a68db083834 | cirros-0.3.4-x86_64-uec-kernel  |
    | 0a6c800e-6002-4f2f-a2b9-0542e0f2fac0 | cirros-0.3.4-x86_64-uec-ramdisk |
    | fc0d2030-1cec-4780-9199-e99c63da0005 | test_bk                         |
    | a9d7dc78-acab-4e4e-b03d-c7da9cb5ef00 | test_bk                         |
    | 937061b4-51c7-4835-b9b7-d63464665a27 | test_bk                         |
    | 41e25c26-c61e-421c-b170-a39539bcbc78 | test_bk                         |
    | a15ae46a-a3c3-49a8-be75-1aee724fa4e9 | test_bk                         |
    +--------------------------------------+---------------------------------+

    
   4. Add debug point using pdb.set_trace() in nova.compute.manager._rotate_backups method [1]
    
   5. Delete one of the backup images from above list
   
    $ glance image-delete fc0d2030-1cec-4780-9199-e99c63da0005

   6. Create the insatnce backup by passing the rotation parameter value as 1.
    
    $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 1
   
   7. Checked the n-cpu logs
   
  2016-10-17 15:59:38.978 ERROR oslo_messaging.rpc.server [req-99fcf837-514a-4a80-a19e-c8ac2f26768f admin admin] Exception during message handling
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 179, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     instance=instance)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3001, in backup_instance
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self._rotate_backups(context, instance, backup_type, rotation)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3145, in _rotate_backups
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.image_api.delete(context, image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/api.py", line 141, in delete
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return session.delete(context, image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/glance.py", line 765, in delete
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     raise exception.ImageNotFound(image_id=image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server ImageNotFound: Image fc0d2030-1cec-4780-9199-e99c63da0005 could not be found.
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server
   

  [1]
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3139

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1634773/+subscriptions
Follow ups

[Bug 1634773] Re: backup instance won't be able to delete all images exceeding rotation limit, if user mistakenly deletes one of the backup image
From: Matt Riedemann, 2017-02-10
[Bug 1634773] Re: backup instance won't be able to delete all images exceeding rotation limit, if user mistakenly deletes one of the backup image
From: OpenStack Infra, 2016-12-12