yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

** Also affects: nova/newton
   Importance: Undecided
       Status: New

** Changed in: nova/newton
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova/newton
       Status: New => In Progress

** Changed in: nova/newton
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1634773

Title:
  backup instance won't be able to delete all images exceeding rotation
  limit, if user mistakenly deletes one of the backup image

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  In Progress

Bug description:
  If user have already backed up instance to few numbers and then execute
  backup api with rotation 1, then nova will delete the previously
  created images exceeding rotation limit.

  During deleting these images, if user mistakenly deletes one of the
  image in advance, then backup instance won't be able to delete all
  images exceeding rotation limit causing api failure.

  Steps to reproduce:

   1. Create instance
   
    $ nova boot --flavor <flavor_id> --image <image_id> <server_name>
   
    $ nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec test
    
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+
    | ID                                   | Name | Status | Task State | Power State | Networks                        |
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+
    | a9e41bc5-516c-48f1-bed8-2475650835c8 | test | ACTIVE | -          | Running     | public=2001:db8::8, 172.24.4.12 |
    +--------------------------------------+------+--------+------------+-------------+---------------------------------+

    
   2. Create 5 instance backup using the backup api(hit below command 5 times)
   
    $ nova backup <server> <name> <backup-type> <rotation>
   
    $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 5

   3. Take list of backup images
   
    $ glance image-list
    
    +--------------------------------------+---------------------------------+
    | ID                                   | Name                            |
    +--------------------------------------+---------------------------------+
    | c33790be-df8b-4740-8311-0fa787bfb087 | cirros-0.3.4-x86_64-uec         |
    | de43a375-6d6d-4a30-856e-7a68db083834 | cirros-0.3.4-x86_64-uec-kernel  |
    | 0a6c800e-6002-4f2f-a2b9-0542e0f2fac0 | cirros-0.3.4-x86_64-uec-ramdisk |
    | fc0d2030-1cec-4780-9199-e99c63da0005 | test_bk                         |
    | a9d7dc78-acab-4e4e-b03d-c7da9cb5ef00 | test_bk                         |
    | 937061b4-51c7-4835-b9b7-d63464665a27 | test_bk                         |
    | 41e25c26-c61e-421c-b170-a39539bcbc78 | test_bk                         |
    | a15ae46a-a3c3-49a8-be75-1aee724fa4e9 | test_bk                         |
    +--------------------------------------+---------------------------------+

    
   4. Add debug point using pdb.set_trace() in nova.compute.manager._rotate_backups method [1]
    
   5. Delete one of the backup images from above list
   
    $ glance image-delete fc0d2030-1cec-4780-9199-e99c63da0005

   6. Create the insatnce backup by passing the rotation parameter value as 1.
    
    $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 1
   
   7. Checked the n-cpu logs
   
  2016-10-17 15:59:38.978 ERROR oslo_messaging.rpc.server [req-99fcf837-514a-4a80-a19e-c8ac2f26768f admin admin] Exception during message handling
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     function_name, call_dict, binary)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 179, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     instance=instance)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3001, in backup_instance
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self._rotate_backups(context, instance, backup_type, rotation)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.force_reraise()
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3145, in _rotate_backups
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     self.image_api.delete(context, image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/api.py", line 141, in delete
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     return session.delete(context, image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/image/glance.py", line 765, in delete
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server     raise exception.ImageNotFound(image_id=image_id)
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server ImageNotFound: Image fc0d2030-1cec-4780-9199-e99c63da0005 could not be found.
  2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server
   

  [1]
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3139

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1634773/+subscriptions