yahoo-eng-team team mailing list archive

[Brian Rosmaita <1808456@xxxxxxxxxxxxxxxxxx>]

** Changed in: glance-store
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1808456

Title:
  ceph backend reporting meaningless error when no space left

Status in Glance:
  Invalid
Status in glance_store:
  Fix Released

Bug description:
  When uploading image, but there's no space left in ceph(rbd) backend, client(such as glanceclient) will receive a meaningless error:
  500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

  
  steps to reproduce:
  -------------------------------------------------
  1.Prepare ceph backend for glance, make the free space small enough, e.g. 10MB.
  To be simple, you also can modify ceph's code(function resize), to let it raise errno.ENOSPC. I did this way.
  2.uploading image: glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c

  
  logs
  -------------------------------------------------
  stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
  > /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
  -> backend = None
  (Pdb) c
  [=============================>] 100%
  +------------------+--------------------------------------+
  | Property         | Value                                |
  +------------------+--------------------------------------+
  | checksum         | None                                 |
  | container_format | bare                                 |
  | created_at       | 2018-12-14T02:08:36Z                 |
  | disk_format      | raw                                  |
  | id               | 8c2e48f0-aafc-4744-95b6-fe0b6fbfe975 |
  | min_disk         | 0                                    |
  | min_ram          | 0                                    |
  | name             | img2-ceph                            |
  | os_hash_algo     | None                                 |
  | os_hash_value    | None                                 |
  | os_hidden        | False                                |
  | owner            | 3242a198f7044fcd9b756866ec296391     |
  | protected        | False                                |
  | size             | None                                 |
  | status           | queued                               |
  | tags             | []                                   |
  | updated_at       | 2018-12-14T02:08:36Z                 |
  | virtual_size     | Not available                        |
  | visibility       | public                               |
  +------------------+--------------------------------------+
  500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

  
  expected
  -------------------------------------------------
  The correct message should be something related "Storage Full", rbd.py should raise glance_store.StorageFull, and this exception will be caught by notifier.py.

  Some code snippet in notifier.py:
          except glance_store.StorageFull as e:
              msg = (_("Image storage media is full: %s") %
                     encodeutils.exception_to_unicode(e))
              _send_notification(notify_error, 'image.upload', msg)
              raise webob.exc.HTTPRequestEntityTooLarge(explanation=msg)

  After doing this, the expected behavior will be:
  stack@ubuntu16vmliang:~$ glance image-create --name img2-ceph --visibility public --disk-format raw --container-format bare --progress --backend rbd --file /opt/stack/data/glance/images/d4ca8259-168b-42f5-a719-40038362ae8c
  > /usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py(555)do_image_upload()
  -> backend = None
  (Pdb) c
  [=============================>] 100%
  +------------------+--------------------------------------+
  | Property         | Value                                |
  +------------------+--------------------------------------+
  | checksum         | None                                 |
  | container_format | bare                                 |
  | created_at       | 2018-12-14T01:41:36Z                 |
  | disk_format      | raw                                  |
  | id               | 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 |
  | min_disk         | 0                                    |
  | min_ram          | 0                                    |
  | name             | img2-ceph                            |
  | os_hash_algo     | None                                 |
  | os_hash_value    | None                                 |
  | os_hidden        | False                                |
  | owner            | 3242a198f7044fcd9b756866ec296391     |
  | protected        | False                                |
  | size             | None                                 |
  | status           | queued                               |
  | tags             | []                                   |
  | updated_at       | 2018-12-14T01:41:36Z                 |
  | virtual_size     | Not available                        |
  | visibility       | public                               |
  +------------------+--------------------------------------+
  413 Request Entity Too Large: Image storage media is full: Failed to store image 8aefa92d-bd9c-4726-95ae-d8f698d7bc82 insufficient space available (HTTP 413)

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1808456/+subscriptions