← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #55961

[Bug 1620214] [NEW] Sheepdog: command execution failure

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Sheepdog driver fail to execute 'collie' commands.

How to reproduce:

$ glance image-create --name img00 --disk-format raw --container-format bare --file /tmp/data.img
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | None                                 |
| container_format | bare                                 |
| created_at       | 2016-09-05T07:58:51Z                 |
| disk_format      | raw                                  |
| id               | 66e9827a-e254-4b37-b889-07c6764118d6 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | img00                                |
| owner            | cb433b293d6c498d9a8a3989bb4275ab     |
| protected        | False                                |
| size             | None                                 |
| status           | queued                               |
| tags             | []                                   |
| updated_at       | 2016-09-05T07:58:51Z                 |
| virtual_size     | None                                 |
| visibility       | private                              |
+------------------+--------------------------------------+
500 Internal Server Error
The server has either erred or is incapable of performing the requested operation.
    (HTTP 500)


traceback of g-api:

2016-09-05 07:58:52.207 DEBUG oslo_concurrency.processutils [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] Running cmd (subprocess): collie vdi list -r -a 127.0.0.1 -p 7000 66e9827a-e254-4b37-b889-07c6764118d6  from (pid=17867) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
2016-09-05 07:58:52.222 DEBUG oslo_concurrency.processutils [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] u'collie vdi list -r -a 127.0.0.1 -p 7000 66e9827a-e254-4b37-b889-07c6764118d6 ' failed. Not Retrying. from (pid=17867) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:422                        2016-09-05 07:58:52.224 DEBUG oslo_messaging._drivers.amqpdriver [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] CAST unique_id: fa25c1bf930b44ceb9ff56246e4f2666 NOTIFY exchange 'glance' topic 'notifications.error' from (pid=17867) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432                 2016-09-05 07:58:52.229 ERROR glance.api.v2.image_data [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] Failed to upload image data due to internal error                                                   2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data Traceback (most recent call last):
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/api/v2/image_data.py", line 114, in upload  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     image.set_data(data, size)                                               2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/domain/proxy.py", line 195, in set_data
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.base.set_data(data, size)                                           2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/notifier.py", line 485, in set_data         2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     _send_notification(notify_error, 'image.upload', msg)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__                                                                                                                    2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.force_reraise()
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     six.reraise(self.type_, self.value, self.tb)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/notifier.py", line 432, in set_data
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.repo.set_data(data, size)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/api/policy.py", line 185, in set_data
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     return self.image.set_data(*args, **kwargs)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/quota/__init__.py", line 304, in set_data
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.image.set_data(data, size=size)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/location.py", line 430, in set_data
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier=verifier)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/backend.py", line 447, in add_to_backend
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/backend.py", line 420, in store_add_to_backend
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier=verifier)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/capabilities.py", line 225, in op_checker
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     return store_op_fun(store, *args, **kwargs)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 362, in add
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     if image.exist():
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 200, in exist
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     out = self._run_command("list -r", None)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 137, in _run_command
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     cmd, process_input=data)[0]
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 363, in execute
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     env=env_variables)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 54, in __init__
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     errread, errwrite)
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     raise child_exception
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data OSError: [Errno 2] No such file or directory
2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1620214

Title:
  Sheepdog: command execution failure

Status in Glance:
  New

Bug description:
  Sheepdog driver fail to execute 'collie' commands.

  How to reproduce:

  $ glance image-create --name img00 --disk-format raw --container-format bare --file /tmp/data.img
  +------------------+--------------------------------------+
  | Property         | Value                                |
  +------------------+--------------------------------------+
  | checksum         | None                                 |
  | container_format | bare                                 |
  | created_at       | 2016-09-05T07:58:51Z                 |
  | disk_format      | raw                                  |
  | id               | 66e9827a-e254-4b37-b889-07c6764118d6 |
  | min_disk         | 0                                    |
  | min_ram          | 0                                    |
  | name             | img00                                |
  | owner            | cb433b293d6c498d9a8a3989bb4275ab     |
  | protected        | False                                |
  | size             | None                                 |
  | status           | queued                               |
  | tags             | []                                   |
  | updated_at       | 2016-09-05T07:58:51Z                 |
  | virtual_size     | None                                 |
  | visibility       | private                              |
  +------------------+--------------------------------------+
  500 Internal Server Error
  The server has either erred or is incapable of performing the requested operation.
      (HTTP 500)


  traceback of g-api:

  2016-09-05 07:58:52.207 DEBUG oslo_concurrency.processutils [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] Running cmd (subprocess): collie vdi list -r -a 127.0.0.1 -p 7000 66e9827a-e254-4b37-b889-07c6764118d6  from (pid=17867) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
  2016-09-05 07:58:52.222 DEBUG oslo_concurrency.processutils [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] u'collie vdi list -r -a 127.0.0.1 -p 7000 66e9827a-e254-4b37-b889-07c6764118d6 ' failed. Not Retrying. from (pid=17867) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:422                        2016-09-05 07:58:52.224 DEBUG oslo_messaging._drivers.amqpdriver [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] CAST unique_id: fa25c1bf930b44ceb9ff56246e4f2666 NOTIFY exchange 'glance' topic 'notifications.error' from (pid=17867) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432                 2016-09-05 07:58:52.229 ERROR glance.api.v2.image_data [req-6e1899f4-5c47-4d88-9a08-cbf8a7a67e0c de40eaf63efe4b3ea8d9c83b97482736 cb433b293d6c498d9a8a3989bb4275ab] Failed to upload image data due to internal error                                                   2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data Traceback (most recent call last):
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/api/v2/image_data.py", line 114, in upload  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     image.set_data(data, size)                                               2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/domain/proxy.py", line 195, in set_data
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.base.set_data(data, size)                                           2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/notifier.py", line 485, in set_data         2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     _send_notification(notify_error, 'image.upload', msg)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__                                                                                                                    2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.force_reraise()
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     six.reraise(self.type_, self.value, self.tb)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/notifier.py", line 432, in set_data
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.repo.set_data(data, size)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/api/policy.py", line 185, in set_data
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     return self.image.set_data(*args, **kwargs)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/quota/__init__.py", line 304, in set_data
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     self.image.set_data(data, size=size)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance/glance/location.py", line 430, in set_data
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier=verifier)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/backend.py", line 447, in add_to_backend
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/backend.py", line 420, in store_add_to_backend
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     verifier=verifier)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/capabilities.py", line 225, in op_checker
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     return store_op_fun(store, *args, **kwargs)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 362, in add
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     if image.exist():
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 200, in exist
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     out = self._run_command("list -r", None)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/opt/stack/glance_store/glance_store/_drivers/sheepdog.py", line 137, in _run_command
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     cmd, process_input=data)[0]
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 363, in execute
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     env=env_variables)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/local/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 54, in __init__
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     errread, errwrite)
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data     raise child_exception
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data OSError: [Errno 2] No such file or directory
  2016-09-05 07:58:52.229 TRACE glance.api.v2.image_data

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1620214/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next