yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1700926] Re: Exception Error logs shown in Citrix XenServer CI

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1700926@xxxxxxxxxxxxxxxxxx>
Date: Wed, 20 Sep 2017 22:15:14 -0000
Reply-to: Bug 1700926 <1700926@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/478437
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aaddd51d6fa8863a505e3f33810660b0b27be14d
Submitter: Jenkins
Branch:    master

commit aaddd51d6fa8863a505e3f33810660b0b27be14d
Author: naichuans <naichuan.sun@xxxxxxxxxx>
Date:   Wed Jun 28 08:36:29 2017 +0000

    xenapi: Exception Error logs shown in Citrix XenServer CI
    
    There are many errors in the patchs which passed our XenServer CI,
    it is because when run tempest, test case will ask nova sevice to
    do snapshotting, then delete image before snapshotting done. Then
    nova compute service couldn't find the image when do image upload
    and raise an exception.
    Normally the exception can be consumed by nova service. But the
    exception generated by os-xenapi can't be recognized by nova, so
    the error logs come out.
    Add a exception which computer service can process.
    Modify vmops.py to transfer exception to Nova compute.
    Add unit test for exception transfer
    
    Change-Id: I49f1a2aab507cc4b1c4a3d6c13df96ee3e678a99
    Depends-On: I16d199e93f615e4f374d49948dceeb4d1d600aa9
    Depends-On: I80007bb897a1f7eda47ebda82e08fe270f97ff40
    Closes-Bug: 1700926


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1700926

Title:
  Exception Error logs shown in Citrix XenServer CI

Status in OpenStack Compute (nova):
  Fix Released
Status in os-xenapi:
  Fix Released

Bug description:
  From a patch https://review.openstack.org/#/c/459485/ which passed our XenServer CI, we can see there are some errors in nova http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/85/459485/4/check/dsvm-tempest-neutron-network/61927a4/logs/screen-n-cpu.txt.gz
  We need to figure out whether this is real problem or just configuration problem
  2017-04-25 03:41:04.891 5135 DEBUG nova.virt.xenapi.vm_utils [req-600bd486-ca26-4a98-bcae-9ed50cea1f6b tempest-ImagesOneServerNegativeTestJSON-218904050 tempest-ImagesOneServerNegativeTestJSON-218904050] VHD 00e1b23b-9926-477a-a766-6e792d1a435d has parent 5f75df8a-b35f-447b-a731-7b1e12297271 _get_vhd_parent_uuid /opt/stack/new/nova/nova/virt/xenapi/vm_utils.py:1979
  2017-04-25 03:41:04.895 5135 DEBUG nova.virt.xenapi.vm_utils [req-600bd486-ca26-4a98-bcae-9ed50cea1f6b tempest-ImagesOneServerNegativeTestJSON-218904050 tempest-ImagesOneServerNegativeTestJSON-218904050] VHD 5f75df8a-b35f-447b-a731-7b1e12297271 has parent 1afecff1-7ee0-405a-9f87-5e2b674657b2 _get_vhd_parent_uuid /opt/stack/new/nova/nova/virt/xenapi/vm_utils.py:1979
  2017-04-25 03:41:04.988 5135 DEBUG os_xenapi.client.session [req-600bd486-ca26-4a98-bcae-9ed50cea1f6b tempest-ImagesOneServerNegativeTestJSON-218904050 tempest-ImagesOneServerNegativeTestJSON-218904050] glance.py.upload_vhd2 attempt 1/1, callback_result: http://192.168.33.1:9292 call_plugin_serialized_with_retry /opt/stack/new/os-xenapi/os_xenapi/client/session.py:242
  2017-04-25 03:41:05.233 5135 DEBUG os_xenapi.client.session [req-600bd486-ca26-4a98-bcae-9ed50cea1f6b tempest-ImagesOneServerNegativeTestJSON-218904050 tempest-ImagesOneServerNegativeTestJSON-218904050] Got exception: ['XENAPI_PLUGIN_FAILURE', 'upload_vhd2', 'PluginError', 'Got Permanent Error response [404] while uploading image [244e93c0-28f1-4951-b922-665908bef7d5] to glance http://192.168.33.1:9292/v2/images/244e93c0-28f1-4951-b922-665908bef7d5/file'] _unwrap_plugin_exceptions /opt/stack/new/os-xenapi/os_xenapi/client/session.py:295
  2017-04-25 03:41:05.448 5135 DEBUG nova.compute.manager [req-600bd486-ca26-4a98-bcae-9ed50cea1f6b tempest-ImagesOneServerNegativeTestJSON-218904050 tempest-ImagesOneServerNegativeTestJSON-218904050] [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] Cleaning up image 244e93c0-28f1-4951-b922-665908bef7d5 decorated_function /opt/stack/new/nova/nova/compute/manager.py:235
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] Traceback (most recent call last):
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/compute/manager.py", line 231, in decorated_function
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] *args, **kwargs)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/compute/manager.py", line 3128, in snapshot_instance
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] task_states.IMAGE_SNAPSHOT)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/compute/manager.py", line 3160, in _snapshot_instance
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] update_task_state)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/virt/xenapi/driver.py", line 198, in snapshot
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] self._vmops.snapshot(context, instance, image_id, update_task_state)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/virt/xenapi/vmops.py", line 928, in snapshot
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] vdi_uuids,
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/virt/xenapi/image/glance.py", line 94, in upload_image
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] 'upload_vhd2', params)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/nova/nova/virt/xenapi/image/glance.py", line 56, in _call_glance_plugin
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] **params)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/session.py", line 243, in call_plugin_serialized_with_retry
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] return self.call_plugin_serialized(plugin, fn, *args, **kwargs)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/session.py", line 218, in call_plugin_serialized
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] rv = self.call_plugin(plugin, fn, params)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/session.py", line 214, in call_plugin
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] self.host_ref, plugin, fn, args)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/session.py", line 293, in _unwrap_plugin_exceptions
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] return func(*args, **kwargs)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/XenAPI.py", line 239, in _call_
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] return self._send(self._name, args)
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/XenAPI.py", line 130, in xenapi_request
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] result = _parse_result(getattr(self, methodname)(*full_params))
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] File "/opt/stack/new/os-xenapi/os_xenapi/client/XenAPI.py", line 212, in _parse_result
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] raise Failure(result['ErrorDescription'])
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5] Failure: ['XENAPI_PLUGIN_FAILURE', 'upload_vhd2', 'PluginError', 'Got Permanent Error response [404] while uploading image [244e93c0-28f1-4951-b922-665908bef7d5] to glance http://192.168.33.1:9292/v2/images/244e93c0-28f1-4951-b922-665908bef7d5/file']
  2017-04-25 03:41:05.448 5135 ERROR nova.compute.manager [instance: 92c76bb9-8dfb-4b35-9ee4-137ec23b82d5]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1700926/+subscriptions