yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1803717] [NEW] Instance snapshot fails with rbd backend

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Dr. Jens Harbott" <j.harbott@xxxxxxxx>
Date: Fri, 16 Nov 2018 14:15:49 -0000
Reply-to: Bug 1803717 <1803717@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/85/617985/1/check/devstack-plugin-ceph-
tempest/58fe872/controller/logs/screen-n-cpu.txt.gz#_Nov_16_07_59_55_423217

Nov 16 08:07:14.891163 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG nova.virt.libvirt.storage.rbd_utils [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] creating snapshot(snap) on rbd image(0ef68017-c94d-43b4-8bb9-78f4d77cf928) {{(pid=3629) create_snap /opt/stack/nova/nova/virt/libvirt/storage/rbd_utils.py:383}}
Nov 16 08:07:16.213304 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG oslo_service.periodic_task [None req-898d2dca-37a7-403f-b578-5ca2ae90e329 None None] Running periodic task ComputeManager._cleanup_expired_console_auth_tokens {{(pid=3629) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:219}}
Nov 16 08:07:16.322727 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] Failed to snapshot image: TypeError: add_location() takes exactly 4 arguments (3 given)
Nov 16 08:07:16.322893 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Nov 16 08:07:16.323039 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1908, in snapshot
Nov 16 08:07:16.323192 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     purge_props=False)
Nov 16 08:07:16.323326 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/api.py", line 142, in update
Nov 16 08:07:16.323460 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     purge_props=purge_props)
Nov 16 08:07:16.323604 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 588, in update
Nov 16 08:07:16.323801 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     _reraise_translated_image_exception(image_id)
Nov 16 08:07:16.324000 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 908, in _reraise_translated_image_exception
Nov 16 08:07:16.324179 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     six.reraise(type(new_exc), new_exc, exc_trace)
Nov 16 08:07:16.324362 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 586, in update
Nov 16 08:07:16.324511 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     image = self._update_v2(context, sent_service_image_meta, data)
Nov 16 08:07:16.324655 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 600, in _update_v2
Nov 16 08:07:16.324802 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     image = self._add_location(context, image_id, location)
Nov 16 08:07:16.324948 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 485, in _add_location
Nov 16 08:07:16.325110 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     context, 2, 'add_location', args=(image_id, location))
Nov 16 08:07:16.325263 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 193, in call
Nov 16 08:07:16.325421 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     result = getattr(controller, method)(*args, **kwargs)
Nov 16 08:07:16.325557 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver TypeError: add_location() takes exactly 4 arguments (3 given)
Nov 16 08:07:16.325747 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver 
Nov 16 08:07:16.432786 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG nova.virt.libvirt.storage.rbd_utils [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] removing snapshot(snap) on rbd image(0ef68017-c94d-43b4-8bb9-78f4d77cf928) {{(pid=3629) remove_snap /opt/stack/nova/nova/virt/libvirt/storage/rbd_utils.py:410}}

This error may have been introduced up to three weeks ago without
getting noticed because the ceph job has been broken.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1803717

Title:
  Instance snapshot fails with rbd backend

Status in OpenStack Compute (nova):
  New

Bug description:
  http://logs.openstack.org/85/617985/1/check/devstack-plugin-ceph-
  tempest/58fe872/controller/logs/screen-n-cpu.txt.gz#_Nov_16_07_59_55_423217

  Nov 16 08:07:14.891163 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG nova.virt.libvirt.storage.rbd_utils [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] creating snapshot(snap) on rbd image(0ef68017-c94d-43b4-8bb9-78f4d77cf928) {{(pid=3629) create_snap /opt/stack/nova/nova/virt/libvirt/storage/rbd_utils.py:383}}
  Nov 16 08:07:16.213304 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG oslo_service.periodic_task [None req-898d2dca-37a7-403f-b578-5ca2ae90e329 None None] Running periodic task ComputeManager._cleanup_expired_console_auth_tokens {{(pid=3629) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:219}}
  Nov 16 08:07:16.322727 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] Failed to snapshot image: TypeError: add_location() takes exactly 4 arguments (3 given)
  Nov 16 08:07:16.322893 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
  Nov 16 08:07:16.323039 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1908, in snapshot
  Nov 16 08:07:16.323192 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     purge_props=False)
  Nov 16 08:07:16.323326 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/api.py", line 142, in update
  Nov 16 08:07:16.323460 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     purge_props=purge_props)
  Nov 16 08:07:16.323604 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 588, in update
  Nov 16 08:07:16.323801 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     _reraise_translated_image_exception(image_id)
  Nov 16 08:07:16.324000 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 908, in _reraise_translated_image_exception
  Nov 16 08:07:16.324179 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     six.reraise(type(new_exc), new_exc, exc_trace)
  Nov 16 08:07:16.324362 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 586, in update
  Nov 16 08:07:16.324511 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     image = self._update_v2(context, sent_service_image_meta, data)
  Nov 16 08:07:16.324655 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 600, in _update_v2
  Nov 16 08:07:16.324802 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     image = self._add_location(context, image_id, location)
  Nov 16 08:07:16.324948 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 485, in _add_location
  Nov 16 08:07:16.325110 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     context, 2, 'add_location', args=(image_id, location))
  Nov 16 08:07:16.325263 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver   File "/opt/stack/nova/nova/image/glance.py", line 193, in call
  Nov 16 08:07:16.325421 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver     result = getattr(controller, method)(*args, **kwargs)
  Nov 16 08:07:16.325557 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver TypeError: add_location() takes exactly 4 arguments (3 given)
  Nov 16 08:07:16.325747 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: ERROR nova.virt.libvirt.driver 
  Nov 16 08:07:16.432786 ubuntu-xenial-rax-iad-0000536097 nova-compute[3629]: DEBUG nova.virt.libvirt.storage.rbd_utils [None req-3005471d-96d3-4fdd-a042-0b9e6025ccf4 tempest-ServerActionsTestJSON-406716108 tempest-ServerActionsTestJSON-406716108] removing snapshot(snap) on rbd image(0ef68017-c94d-43b4-8bb9-78f4d77cf928) {{(pid=3629) remove_snap /opt/stack/nova/nova/virt/libvirt/storage/rbd_utils.py:410}}

  This error may have been introduced up to three weeks ago without
  getting noticed because the ceph job has been broken.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1803717/+subscriptions
Follow ups

[Bug 1803717] Re: Instance snapshot fails with rbd backend
From: Matt Riedemann, 2019-04-23
[Bug 1803717] Re: Instance snapshot fails with rbd backend
From: OpenStack Infra, 2018-11-19