yahoo-eng-team team mailing list archive

[Gaël THEROND <1860739@xxxxxxxxxxxxxxxxxx>]

** Changed in: cinder
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860739

Title:
  Cinder backup failed for restore volume snapshot (for instance
  snapshot)

Status in Cinder:
  Fix Released
Status in kolla-ansible:
  Invalid
Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Cinder backup failed for restore volume snapshot.
  Works for restore volumes without snapshots.
  I use a external ceph cluster version 14.2.6 nautilus (stable)

  
  What you expected to happen:
  Restore volume snapshot backup.

  How to reproduce it (minimal and precise):
  - Create a instance with disk;
  - Snaptshot instance;
  - Backup volume snapshot;
  Example: openstack volume backup create 18363c93-e577-470e-971a-678094bce943 --snapshot 1624b162-e616-47dc-bb1b-01fe4843c581

  - Restore backup;

  CentOS Linux release 7.7.1908 (Core)
  3.10.0-1062.9.1.el7.x86_64
  Docker 19.03.5
  Docker image source
  Docker image distribution Centos
  official images from Docker Hub
  git status: # On branch stable/train

  
  ==> /var/log/kolla/cinder/cinder-backup.log <==
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server [req-d37e8900-4c9b-45d7-9b6b-59dd66159e86 0af9f5deabf74f4e8efea750cb3c7cf0 1baa170cb3a742e59db2e7e5948d6e7f - default default] Exception during message handling: ImageNotFound: [errno 2] error opening image volume-18363c93-e577-470e-971a-678094bce943.backup.base at snapshot None
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 608, in restore_backup
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     backup.save()
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 597, in restore_backup
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     self._run_restore(context, backup, volume)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 659, in _run_restore
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     tpool.Proxy(device_path))
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1264, in restore
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     self._restore_volume(backup, target_volume, volume_file)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1231, in _restore_volume
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     length, src_snap=restore_point)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1026, in _full_restore
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server     read_only=True))
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2753, in rbd.Image.__init__ (/builddir/build/BUILD/ceph-14.2.1/build/src/pybind/rbd/pyrex/rbd.c:24354)
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server ImageNotFound: [errno 2] error opening image volume-18363c93-e577-470e-971a-678094bce943.backup.base at snapshot None
  2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server

  # cat /etc/kolla/config/cinder/cinder-volume.conf
  [DEFAULT]
  enabled_backends=netapp06,rbd-1
  debug = True

  [rbd-1]
  rbd_ceph_conf=/etc/ceph/ceph.conf
  rbd_user=cinder
  backend_host=rbd:volumes
  rbd_pool=volumes
  volume_backend_name=rbd-1
  volume_driver=cinder.volume.drivers.rbd.RBDDriver
  rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}

  [netapp06]
  volume_backend_name=netapp06
  nfs_shares_config=/etc/cinder/nfs_shares
  volume_driver=cinder.volume.drivers.nfs.NfsDriver
  nfs_snapshot_support = True
  nas_secure_file_operations = False
  image_volume_cache_enabled = True

  # cat /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder-backup.keyring
  [client.cinder-backup]
          key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  (kolla) [root@kolla-deploy config]# cat /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
  [client.cinder]
          key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  (kolla) [root@kolla-deploy config]#

  # cat /etc/kolla/config/cinder/cinder-backup.conf
  [DEFAULT]
  backup_ceph_conf=/etc/ceph/ceph.conf
  backup_ceph_user=cinder-backup
  backup_ceph_chunk_size = 134217728
  backup_ceph_pool=backups
  backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
  backup_ceph_stripe_unit = 0
  backup_ceph_stripe_count = 0
  restore_discard_excess_bytes = true
  (kolla) [root@kolla-deploy config]#

  # cat /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
  [client.cinder]
          key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  # cat /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring
  [client.cinder-backup]
          key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  (kolla) [root@kolla-deploy config]#

  # grep ^[^#] /etc/kolla/globals.yml
  ---
  kolla_install_type: "source"
  openstack_release: "train"
  kolla_internal_vip_address: "172.16.120.154"
  kolla_external_vip_address: "XXX.XXX.XXX.XXX"
  kolla_external_fqdn: "cloud.xxxxxx.com.br"
  kolla_enable_tls_external: "yes"
  kolla_external_fqdn_cert: "{{ node_config }}/certificates/xxxxxx.pem"
  kolla_external_fqdn_cacert: "{{ node_config }}/certificates/xxxxxx-ca.crt"
  enable_cinder: "yes"
  enable_cinder_backup: "yes"
  enable_mariabackup: "yes"
  enable_neutron_vpnaas: "yes"
  enable_neutron_agent_ha: "yes"
  glance_backend_ceph: "yes"
  glance_backend_file: "no"
  cinder_backend_ceph: "yes"
  nova_backend_ceph: "yes"

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1860739/+subscriptions