yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1486032] [NEW] Swapping a volume cause an exception

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Fabien Dupont <fabien.dupont@xxxxxxxxxxxx>
Date: Tue, 18 Aug 2015 13:08:03 -0000
Reply-to: Bug 1486032 <1486032@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi,

Apparently, when cinder tells nova to swap the volume a of VM after a
retyping it, this exception appears in the logs and the volume is not
swapped. It seems the instance UUID is passed as parameter instead of a
dict.


I'm using last kilo packaged version for ubuntu 14.014 :

root@cmp-00004-nivol:~# dpkg -l | grep nova
ii  nova-common                         1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - common files
ii  nova-compute                        1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node base
ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node libvirt support
ii  python-nova                         1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

Here are some logs :

2015-08-18 14:53:11.599 577 ERROR root [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 346, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4946, in swap_volume\n    context, old_volume_id, instance_uuid=instance.uuid)\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager Traceback (most recent call last):
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     return function(self, context, *args, **kwargs)
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 358, in decorated_function
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     kwargs['instance'], e, sys.exc_info())
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 87, in add_instance_fault_from_exc
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     fault_obj.instance_uuid = instance.uuid
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager AttributeError: 'unicode' object has no attribute 'uuid'
2015-08-18 14:53:11.601 577 TRACE nova.compute.manager 
2015-08-18 14:53:11.605 577 ERROR root [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 358, in decorated_function\n    kwargs[\'instance\'], e, sys.exc_info())\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 87, in add_instance_fault_from_exc\n    fault_obj.instance_uuid = instance.uuid\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
2015-08-18 14:53:11.606 577 ERROR oslo_messaging.rpc.dispatcher [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Exception during message handling: string indices must be integers
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6775, in swap_volume
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 317, in decorated_function
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     instance_uuid = keyed_args['instance']['uuid']
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher TypeError: string indices must be integers
2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher 

To reproduce this, just create a volume, retype it while the VM is
running and wait for the exception.

Thx,
\fab

** Affects: cinder
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New

** Also affects: cinder
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1486032

Title:
  Swapping a volume cause an exception

Status in Cinder:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  Hi,

  Apparently, when cinder tells nova to swap the volume a of VM after a
  retyping it, this exception appears in the logs and the volume is not
  swapped. It seems the instance UUID is passed as parameter instead of
  a dict.

  
  I'm using last kilo packaged version for ubuntu 14.014 :

  root@cmp-00004-nivol:~# dpkg -l | grep nova
  ii  nova-common                         1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1.0-0ubuntu1.1~cloud0          all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  Here are some logs :

  2015-08-18 14:53:11.599 577 ERROR root [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 346, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4946, in swap_volume\n    context, old_volume_id, instance_uuid=instance.uuid)\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager Traceback (most recent call last):
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     return function(self, context, *args, **kwargs)
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 358, in decorated_function
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     kwargs['instance'], e, sys.exc_info())
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 87, in add_instance_fault_from_exc
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager     fault_obj.instance_uuid = instance.uuid
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager AttributeError: 'unicode' object has no attribute 'uuid'
  2015-08-18 14:53:11.601 577 TRACE nova.compute.manager 
  2015-08-18 14:53:11.605 577 ERROR root [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 358, in decorated_function\n    kwargs[\'instance\'], e, sys.exc_info())\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 87, in add_instance_fault_from_exc\n    fault_obj.instance_uuid = instance.uuid\n', "AttributeError: 'unicode' object has no attribute 'uuid'\n"]
  2015-08-18 14:53:11.606 577 ERROR oslo_messaging.rpc.dispatcher [req-fa61a8e5-e8a7-46ef-b7ec-5d15129cada0 e5ea8d8e42e443f0963b2cd8288a6880 de0701822aa446f6b1fd9a2c8757b1b6 - - -] Exception during message handling: string indices must be integers
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6775, in swap_volume
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     new_volume_id)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 317, in decorated_function
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher     instance_uuid = keyed_args['instance']['uuid']
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher TypeError: string indices must be integers
  2015-08-18 14:53:11.606 577 TRACE oslo_messaging.rpc.dispatcher 

  To reproduce this, just create a volume, retype it while the VM is
  running and wait for the exception.

  Thx,
  \fab

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1486032/+subscriptions
Follow ups

[Bug 1486032] Re: Swapping a volume cause an exception
From: Takashi NATSUME, 2017-06-27
[Bug 1486032] Re: Swapping a volume cause an exception
From: Eric Harney, 2015-09-03