yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1498645] [NEW] [VMware] Creating snapshot stuck at "Image uploading" state

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Giridhar Jayavelu <gjayavelu@xxxxxxxxxx>
Date: Tue, 22 Sep 2015 20:35:18 -0000
Reply-to: Bug 1498645 <1498645@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

- Started devstack from master branch
- Created an instance which is backed by iscsi datastore on vsphere
Creating snapshot got stuck at "Image Uploading" state.

Noticed "Operation timed out" error while executing "Export OVF
template" task.

vpxd.log
------------
2015-09-22T20:25:49.681Z info vpxd[7F5A52B6A700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12119-ngc-8] [VpxLRO] -- BEGIN task-internal-209954 -- datastore-60 -- vim.Datastore.GetBrowser -- 52a48e8a-5d02-e8d3-f168-22a925987f0b(52100a84-20a1-b773-bec2-1e9f6ab7c5a4)
2015-09-22T20:25:49.682Z info vpxd[7F5A52B6A700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12119-ngc-8] [VpxLRO] -- FINISH task-internal-209954
2015-09-22T20:25:49.686Z info vpxd[7F5A52C6C700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12120-ngc-b4] [VpxLRO] -- BEGIN session[52a48e8a-5d02-e8d3-f168-22a925987f0b]52ff9143-1a53-1738-6c4c-4226b0667c04 -- datastoreBrowser-datastore-60 -- vim.host.DatastoreBrowser.search -- 52a48e8a-5d02-e8d3-f168-22a925987f0b(52100a84-20a1-b773-bec2-1e9f6ab7c5a4)
2015-09-22T20:25:49.806Z info vpxd[7F5A52C6C700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12120-ngc-b4] [VpxLRO] -- FINISH session[52a48e8a-5d02-e8d3-f168-22a925987f0b]52ff9143-1a53-1738-6c4c-4226b0667c04
2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=VAppExport opID=5d311e85-e] [ExportTaskMo] Timed out
2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=VAppExport opID=5d311e85-e] [ExportTaskMo] Task timed out
2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=MoHttpNfcLease opID=5d311e85-e] [HttpNfcLeaseMo] Releasing HTTP-NFC ticket
2015-09-22T20:25:50.649Z warning vpxd[7F5A5AA03700] [Originator@6876 sub=VpxProfiler opID=5d311e85-e] VpxLro::LroMain [TotalTime] took 300023 ms
2015-09-22T20:25:50.649Z info vpxd[7F5A5AA03700] [Originator@6876 sub=vpxLro opID=5d311e85-e] [VpxLRO] -- FINISH task-872
2015-09-22T20:25:50.649Z info vpxd[7F5A5AA03700] [Originator@6876 sub=Default opID=5d311e85-e] [VpxLRO] -- ERROR task-872 -- vm-176 -- VirtualMachine.ExportVmLRO: vim.fault.Timedout:
--> Result:
--> (vim.fault.Timedout) {
-->    faultCause = (vmodl.MethodFault) null,
-->    msg = ""
--> }
--> Args:
-->


g-api.log
-------------
2015-09-22 13:23:33.232 21257 ERROR glance_store._drivers.vmware_datastore [req-d91e76c5-746d-4b0f-8942-cd5adf866b91 86ba66edc4b24e639c37e4ce992d9384 3d5d5d98dde249f08298210cb2e45866 - - -] Communication error sending http PUT requestto the url /folder/openstack_glance/89c4faf4-79d7-4593-9b6d-4ea4827e3f7c%3FdcPath%3Ddatacenter-1%26dsName%3Dshared.
Got IOError [Errno 32] Broken pipe
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils [req-d91e76c5-746d-4b0f-8942-cd5adf866b91 86ba66edc4b24e639c37e4ce992d9384 3d5d5d98dde249f08298210cb2e45866 - - -] Failed to upload image 89c4faf4-79d7-4593-9b6d-4ea4827e3f7c
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils Traceback (most recent call last):
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     context=req.context)
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 340, in store_add_to_backend
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     context=context)
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     return store_op_fun(store, *args, **kwargs)
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 522, in add
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     res = conn.getresponse()
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     response.begin()
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 415, in begin
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     version, status, reason = self._read_status()
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     raise BadStatusLine(line)
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils BadStatusLine: ''
2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils


git commits used

devstack
-------------
commit c00e39901be810deb4044904734cc68af42aad8e
Merge: 7d4485c be65c6f
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Tue Sep 15 03:17:15 2015 +0000

    Merge "Fix typos in stackrc and unstack.sh"

glance
----------
commit 925794c63e38263198345f92124f370211a9d39e
Merge: 84437b9 5c3a3bd
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Tue Sep 15 02:00:35 2015 +0000

    Merge "Make task_time_to_live work"

nova
-------
commit fe1d118c0c37cd94fe05271946ded8c2b11f9482
Merge: b14bc0a 812d75e
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Wed Sep 16 05:13:44 2015 +0000

    Merge "Fix cells use of legacy bdms during local instance delete
operations"

** Affects: glance
     Importance: Undecided
     Assignee: Giridhar Jayavelu (gjayavelu)
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1498645

Title:
  [VMware] Creating snapshot stuck at "Image uploading" state

Status in Glance:
  New

Bug description:
  - Started devstack from master branch
  - Created an instance which is backed by iscsi datastore on vsphere
  Creating snapshot got stuck at "Image Uploading" state.

  Noticed "Operation timed out" error while executing "Export OVF
  template" task.

  vpxd.log
  ------------
  2015-09-22T20:25:49.681Z info vpxd[7F5A52B6A700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12119-ngc-8] [VpxLRO] -- BEGIN task-internal-209954 -- datastore-60 -- vim.Datastore.GetBrowser -- 52a48e8a-5d02-e8d3-f168-22a925987f0b(52100a84-20a1-b773-bec2-1e9f6ab7c5a4)
  2015-09-22T20:25:49.682Z info vpxd[7F5A52B6A700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12119-ngc-8] [VpxLRO] -- FINISH task-internal-209954
  2015-09-22T20:25:49.686Z info vpxd[7F5A52C6C700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12120-ngc-b4] [VpxLRO] -- BEGIN session[52a48e8a-5d02-e8d3-f168-22a925987f0b]52ff9143-1a53-1738-6c4c-4226b0667c04 -- datastoreBrowser-datastore-60 -- vim.host.DatastoreBrowser.search -- 52a48e8a-5d02-e8d3-f168-22a925987f0b(52100a84-20a1-b773-bec2-1e9f6ab7c5a4)
  2015-09-22T20:25:49.806Z info vpxd[7F5A52C6C700] [Originator@6876 sub=vpxLro opID=257026c4-1697-4696-b644-8e8533a3ef91-12120-ngc-b4] [VpxLRO] -- FINISH session[52a48e8a-5d02-e8d3-f168-22a925987f0b]52ff9143-1a53-1738-6c4c-4226b0667c04
  2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=VAppExport opID=5d311e85-e] [ExportTaskMo] Timed out
  2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=VAppExport opID=5d311e85-e] [ExportTaskMo] Task timed out
  2015-09-22T20:25:50.643Z info vpxd[7F5A5AA03700] [Originator@6876 sub=MoHttpNfcLease opID=5d311e85-e] [HttpNfcLeaseMo] Releasing HTTP-NFC ticket
  2015-09-22T20:25:50.649Z warning vpxd[7F5A5AA03700] [Originator@6876 sub=VpxProfiler opID=5d311e85-e] VpxLro::LroMain [TotalTime] took 300023 ms
  2015-09-22T20:25:50.649Z info vpxd[7F5A5AA03700] [Originator@6876 sub=vpxLro opID=5d311e85-e] [VpxLRO] -- FINISH task-872
  2015-09-22T20:25:50.649Z info vpxd[7F5A5AA03700] [Originator@6876 sub=Default opID=5d311e85-e] [VpxLRO] -- ERROR task-872 -- vm-176 -- VirtualMachine.ExportVmLRO: vim.fault.Timedout:
  --> Result:
  --> (vim.fault.Timedout) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    msg = ""
  --> }
  --> Args:
  -->

  
  g-api.log
  -------------
  2015-09-22 13:23:33.232 21257 ERROR glance_store._drivers.vmware_datastore [req-d91e76c5-746d-4b0f-8942-cd5adf866b91 86ba66edc4b24e639c37e4ce992d9384 3d5d5d98dde249f08298210cb2e45866 - - -] Communication error sending http PUT requestto the url /folder/openstack_glance/89c4faf4-79d7-4593-9b6d-4ea4827e3f7c%3FdcPath%3Ddatacenter-1%26dsName%3Dshared.
  Got IOError [Errno 32] Broken pipe
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils [req-d91e76c5-746d-4b0f-8942-cd5adf866b91 86ba66edc4b24e639c37e4ce992d9384 3d5d5d98dde249f08298210cb2e45866 - - -] Failed to upload image 89c4faf4-79d7-4593-9b6d-4ea4827e3f7c
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils Traceback (most recent call last):
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     context=req.context)
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 340, in store_add_to_backend
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     context=context)
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     return store_op_fun(store, *args, **kwargs)
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 522, in add
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     res = conn.getresponse()
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     response.begin()
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 415, in begin
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     version, status, reason = self._read_status()
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils     raise BadStatusLine(line)
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils BadStatusLine: ''
  2015-09-22 13:23:33.233 21257 ERROR glance.api.v1.upload_utils

  
  git commits used

  devstack
  -------------
  commit c00e39901be810deb4044904734cc68af42aad8e
  Merge: 7d4485c be65c6f
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Tue Sep 15 03:17:15 2015 +0000

      Merge "Fix typos in stackrc and unstack.sh"

  glance
  ----------
  commit 925794c63e38263198345f92124f370211a9d39e
  Merge: 84437b9 5c3a3bd
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Tue Sep 15 02:00:35 2015 +0000

      Merge "Make task_time_to_live work"

  nova
  -------
  commit fe1d118c0c37cd94fe05271946ded8c2b11f9482
  Merge: b14bc0a 812d75e
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Wed Sep 16 05:13:44 2015 +0000

      Merge "Fix cells use of legacy bdms during local instance delete
  operations"

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1498645/+subscriptions
Follow ups

[Bug 1498645] Re: [VMware] Creating snapshot stuck at "Image uploading" state
From: Sabari Murugesan, 2015-09-23