yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1580625] Re: volume-backed resize intermittently fails with rbd imagebackend

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Fri, 16 Jun 2017 17:41:49 -0000
Reply-to: Bug 1580625 <1580625@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova/mitaka
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1580625

Title:
  volume-backed resize intermittently fails with rbd imagebackend

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  Fix Released
Status in OpenStack Compute (nova) mitaka series:
  Fix Released

Bug description:
  After this Tempest change landed to test volume-backed resize:

  https://review.openstack.org/#/c/314816/

  The ceph plugin job has been intermittently failing the test:

  http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full-
  devstack-plugin-
  ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874

  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     payload)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     do_confirm_resize(context, instance, migration.id)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 3336, in do_confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     migration=migration)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 3362, in _confirm_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     network_info)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7288, in confirm_migration
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     self._cleanup_resize(instance, network_info)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1061, in _cleanup_resize
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     ignore_errors=True)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 903, in remove_snap
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     driver._disconnect_from_rados(client, ioctx)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     read_only=read_only)
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/rbd.py", line 374, in __init__
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server     raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server ImageNotFound: error opening image 14decc0d-ac5a-4c29-95cd-a8b1bd48e5e8_disk at snapshot None
  2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server 

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ImageNotFound%3A%20error%20opening%20image%5C%22%20AND%20message%3A%5C%22at%20snapshot%20None%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

  So far there are 29 hits in 24 hours since the Tempest change merged,
  and since it's Tempest it's hitting on all branches for any projects
  that run the ceph job, which is at least nova/cinder/glance/glance-
  store/os-brick/gnocchi.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1580625/+subscriptions
References

[Bug 1580625] [NEW] volume-backed resize intermittently fails with rbd imagebackend
From: Matt Riedemann, 2016-05-11