yahoo-eng-team team mailing list archive

[Doug Hellmann <doug@xxxxxxxxxxxxxxxx>]

** Changed in: glance-store
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1401778

Title:
  500 error returned while uploading image using multi filesystem store
  and 'filesystem_store_metadata_file' option enabled

Status in Glance:
  Triaged
Status in glance_store:
  Fix Released

Bug description:
  When we try to upload image with multi filesystem store enabled and 'filesystem_store_metadata_file' 
  (containing list of all directories configured for multi filesystem store) is provided
  then glance throws 'HTTPInternalServerError (HTTP 500)' error.

  - Glance Configuration:

  1. /etc/glance/glance-api.conf

  [DEFAULT]
  show_multiple_locations = True
  filesystem_store_metadata_file = /etc/glance/metadata.json

  [glance_store]
  filesystem_store_datadirs = /var/lib/glance/images1/:1
  filesystem_store_datadirs = /var/lib/glance/images2/:2

  
  2. /etc/glance/metadata.json

  [
      {
          "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
          "mountpoint": "/var/lib/glance/images1/" 
      },
      {
          "id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2",
          "mountpoint": "/var/lib/glance/images2/" 
      }
  ]

  3. 'df -ha' command result:

  openstack@openstack:/opt/stack$ df -ha
  Filesystem                           Size  Used Avail Use% Mounted on
  proc                                    0     0     0    - /proc
  sysfs                                   0     0     0    - /sys
  tmpfs                                799M  600K  798M   1% /run
  /dev/sda1                            236M   41M  183M  19% /boot
  nfsd                                    0     0     0    - /proc/fs/nfsd
  10.69.4.173:/export/images2          443G   23G  398G   6% /var/lib/glance/images2
  10.69.4.172:/export/images1          447G  8.7G  415G   3% /var/lib/glance/images1
  openstack@openstack:/opt/stack$

  
  - Steps to reproduce:

  1. Create image:
  glance --os-image-api-version 2 image-create --name Test123 --disk-format raw --container-format ami

  2. Upload image data:
   
  openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload 47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py

  HTTPInternalServerError (HTTP 500)
  openstack@openstack-150:~$

  
  - glance-api.log :

  2014-12-11 22:16:59.586 3495 ERROR glance_store.backend [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver <glance_store._drivers.filesystem.Store
   object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-
  8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
  2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to internal error
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most recent call last):
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data image.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.base.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 252, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.notifier.error('image.upload', msg)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 201, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 176, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 364, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data context=self.context)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in add_to_backend
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in store_add_to_backend
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise exceptions.BackendException(msg)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException: The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type

  
  When we enable 'filesystem_store_metadata_file' file option in glance-api.conf then it adds this json data as metadata for all glance images.
  This data is then used by NOVA to directly copy image using 'cp' command. (when glance image directories are shared between nova-compute and glance)
  For direct download in nova.conf we need to enable 'allowed_direct_url_schemes' option.
  Also nova needs 'id' and 'moutpoint' keys to be present in glance-image metadata for direct image copy.

  If there is only one glance image directory (ie- one nfs mountpoint) then we do not get this '500 internal server error'.
  But if there are multiple image data directories configured and in metadata JSON file we give list of dicts containing all 
  mountpoints then we get above error.

  IMO we need to validate glance metadata JSON file (for 'id' and
  'mountpoint' keys) before adding it as image-metadata and if there are
  multiple mountpoints (directories) configured for glance then we need
  to add appropriate metadata to glance-image instead of adding all
  image metadata which is provided in JSON file.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1401778/+subscriptions