yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1206773] Re: XENAPI_PLUGIN_FAILURE', 'download_vhd', 'KeyError', "'args'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Louis Taylor <kragniz@xxxxxxxxx>
Date: Mon, 17 Nov 2014 11:30:12 -0000
Reply-to: Bug 1206773 <1206773@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This looks like a problem with xenserver, not glance.

** Changed in: glance
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1206773

Title:
  XENAPI_PLUGIN_FAILURE', 'download_vhd', 'KeyError', "'args'

Status in OpenStack Image Registry and Delivery Service (Glance):
  Invalid

Bug description:
  I meet the below blocked issue when starting a VM with xenserver6.2 and openstack E version: 
  Could you please figure out the root cause of this issue?  Your support is much appreciated.

  
  My test steps is followed : 
  1.	Install one openstack node (all in one with Ubuntu12.04  ) on PV instance of xenserver 6.2 
  2.	Deploy  the xenapi plugin and make other setting on Dom0
  3.	Install python-xenapi on DomU
  4.	Integrate compute service of openstack with xenserver (see attached nova.conf  file)
  5.	Convert cirros-0.3.0-x86_64-disk.img to VHD format and upload to glance like this: 
                glance add name=cirros-0.3.0-x86_64 is_public=true  container_format=ovf \
  disk_format=vhd < tarred.tgz
  6.	Boot a VM with the above image then fail, this error log is as below. 

  
  In nova-compute.log of domU : 

  9ab972f729f1e1fc] Making asynchronous call on network ... from (pid=1246) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:326
  2013-07-31 03:19:04 DEBUG nova.rpc.amqp [req-2639c13b-1cc7-4cc4-b35d-a7190d1fd814 802fe742f0cc45a592ace109696ce597 65da626d19a842ea9ab972f729f1e1fc] MSG_ID is 83c62dadf8814d13bdfac73149dbd3a4 from (pid=1246) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:329
  2013-07-31 03:19:05 ERROR nova.rpc.amqp [req-2639c13b-1cc7-4cc4-b35d-a7190d1fd814 802fe742f0cc45a592ace109696ce597 65da626d19a842ea9ab972f729f1e1fc] Exception during message handling
  2013-07-31 03:19:05 TRACE nova.rpc.amqp Traceback (most recent call last):
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     rval = node_func(context=ctxt, **node_args)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     return f(*args, **kw)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     sys.exc_info())
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self.gen.next()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     return function(self, context, instance_uuid, *args, **kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 676, in run_instance
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     do_run_instance()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 990, in inner
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 675, in do_run_instance
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._run_instance(context, instance_uuid, **kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 476, in _run_instance
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._set_instance_error_state(context, instance_uuid)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self.gen.next()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 457, in _run_instance
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._deallocate_network(context, instance)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self.gen.next()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 454, in _run_instance
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     injected_files, admin_password)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 617, in _spawn
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._legacy_nw_info(network_info), block_device_info)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi_conn.py", line 184, in spawn
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._vmops.spawn(context, instance, image_meta, network_info)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 359, in spawn
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     undo_mgr.rollback_and_reraise(msg=msg)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1773, in rollback_and_reraise
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self._rollback()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     self.gen.next()
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 346, in spawn
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     vdis = create_disks_step(undo_mgr)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 138, in inner
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     rv = f(*args, **kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 265, in create_disks_step
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     vdis = self._create_disks(context, instance, image_meta)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 242, in _create_disks
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     disk_image_type)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 649, in create_image
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     project_id, image_type)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 707, in fetch_image
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     session, instance, image, image_type)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 764, in _fetch_image_glance_vhd
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     vdis = cls._retry_glance_download_vhd(context, session, image)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 738, in _retry_glance_download_vhd
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     result = session.call_plugin('glance', 'download_vhd', kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi_conn.py", line 601, in call_plugin
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     host, plugin, fn, args)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     rv = meth(*args,**kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi_conn.py", line 610, in _unwrap_plugin_exceptions
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     return func(*args, **kwargs)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 235, in __call__
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     return self.__send(self.__name, args)
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 139, in xenapi_request
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     result = _parse_result(getattr(self, methodname)(*full_params))
  2013-07-31 03:19:05 TRACE nova.rpc.amqp   File "/usr/lib/python2.7/dist-packages/XenAPI.py", line 209, in _parse_result
  2013-07-31 03:19:05 TRACE nova.rpc.amqp     raise Failure(result['ErrorDescription'])
  2013-07-31 03:19:05 TRACE nova.rpc.amqp Failure: ['XENAPI_PLUGIN_FAILURE', 'download_vhd', 'KeyError', "'args'"]
  2013-07-31 03:19:05 TRACE nova.rpc.amqp
  2013-07-31 03:19:56 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=1246) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:154

  
  In xensource.log of dom0: 
  Jul 31 10:18:59 dl380-g5-4 xcp-rrdd: [debug|dl380-g5-4|0 monitor|main|rrdd_main] do_read: /opt/xensource/bin/tapdisk-cache-stats e255e8ee-3566-dac0-c07e-85e2ea85ac4b
  Jul 31 10:19:04 dl380-g5-4 xapi: [debug|dl380-g5-4|278807 INET 0.0.0.0:80|host.call_plugin R:f88b30ef67c0|audit] Host.call_plugin host = 'be43557b-f1a6-46f3-85f5-0e4262ca9a0d (dl380-g5-4)'; plugin = 'glance'; fn = 'download_vhd'; args = [ params: (dp1 S'num_retries' p2 I0 sS'auth_token' p3 V3a028fa2ff434c0381cfcff46d5730a5 p4 sS'glance_port' p5 I9292 sS'uuid_stack' p6 (lp7 S'e398d302-3d94-4e84-9e1d-e52a7f3b0750' p8 aS'be137ca2-ced7-4c85-a010-ac1cd5c813ee' p9 aS'74999a8e-25f3-4437-abf1-d648260cb44a' p10 asS'image_id' p11 Vbe183163-7022-4f4b-8ad8-7c0e3840e637 p12 sS'glance_host' p13 S'10.100.3.172' p14 sS'sr_path' p15 S'/var/run/sr-mount/e255e8ee-3566-dac0-c07e-85e2ea85ac4b' p16 s.; host_uuid: be43557b-f1a6-46f3-85f5-0e4262ca9a0d ]
  Jul 31 10:19:04 dl380-g5-4 xapi: [debug|dl380-g5-4|278807 INET 0.0.0.0:80|host.call_plugin R:f88b30ef67c0|backtrace] Raised at xapi_plugins.ml:50.44-83 -> message_forwarding.ml:233.25-44 -> rbac.ml:229.16-23
  Jul 31 10:19:04 dl380-g5-4 xapi: [debug|dl380-g5-4|278807 INET 0.0.0.0:80|host.call_plugin R:f88b30ef67c0|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
  Jul 31 10:19:04 dl380-g5-4 xapi: [debug|dl380-g5-4|278807 INET 0.0.0.0:80|host.call_plugin R:f88b30ef67c0|dispatcher] Server_helpers.exec exception_handler: Got exception XENAPI_PLUGIN_FAILURE: [ download_vhd; KeyError; 'args' ]
  ===========

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1206773/+subscriptions