yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1401778] [NEW] 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Abhijeet Malawade <Abhijeet.Malawade@xxxxxxxxxxx>
Date: Fri, 12 Dec 2014 06:58:33 -0000
Reply-to: Bug 1401778 <1401778@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

When we try to upload image with multi filesystem store enabled and 'filesystem_store_metadata_file' 
(containing list of all directories configured for multi filesystem store) is provided
then glance throws 'HTTPInternalServerError (HTTP 500)' error.

- Glance Configuration:

1. /etc/glance/glance-api.conf

[DEFAULT]
show_multiple_locations = True
filesystem_store_metadata_file = /etc/glance/metadata.json

[glance_store]
filesystem_store_datadirs = /var/lib/glance/images1/:1
filesystem_store_datadirs = /var/lib/glance/images2/:2


2. /etc/glance/metadata.json

[
    {
        "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
        "mountpoint": "/var/lib/glance/images1/" 
    },
    {
        "id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2",
        "mountpoint": "/var/lib/glance/images2/" 
    }
]

3. 'df -ha' command result:

openstack@openstack:/opt/stack$ df -ha
Filesystem                           Size  Used Avail Use% Mounted on
proc                                    0     0     0    - /proc
sysfs                                   0     0     0    - /sys
tmpfs                                799M  600K  798M   1% /run
/dev/sda1                            236M   41M  183M  19% /boot
nfsd                                    0     0     0    - /proc/fs/nfsd
10.69.4.173:/export/images2          443G   23G  398G   6% /var/lib/glance/images2
10.69.4.172:/export/images1          447G  8.7G  415G   3% /var/lib/glance/images1
openstack@openstack:/opt/stack$


- Steps to reproduce:

1. Create image:
glance --os-image-api-version 2 image-create --name Test123 --disk-format raw --container-format ami

2. Upload image data:
 
openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload 47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py

HTTPInternalServerError (HTTP 500)
openstack@openstack-150:~$


- glance-api.log :

2014-12-11 22:16:59.586 3495 ERROR glance_store.backend [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver <glance_store._drivers.filesystem.Store
 object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-
8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to internal error
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most recent call last):
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 252, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.notifier.error('image.upload', msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 201, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 176, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 364, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data context=self.context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in store_add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise exceptions.BackendException(msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException: The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type


When we enable 'filesystem_store_metadata_file' file option in glance-api.conf then it adds this json data as metadata for all glance images.
This data is then used by NOVA to directly copy image using 'cp' command. (when glance image directories are shared between nova-compute and glance)
For direct download in nova.conf we need to enable 'allowed_direct_url_schemes' option.
Also nova needs 'id' and 'moutpoint' keys to be present in glance-image metadata for direct image copy.

If there is only one glance image directory (ie- one nfs mountpoint) then we do not get this '500 internal server error'.
But if there are multiple image data directories configured and in metadata JSON file we give list of dicts containing all 
mountpoints then we get above error.

IMO we need to validate glance metadata JSON file (for 'id' and
'mountpoint' keys) before adding it as image-metadata and if there are
multiple mountpoints (directories) configured for glance then we need to
add appropriate metadata to glance-image instead of adding all image
metadata which is provided in JSON file.

** Affects: glance
     Importance: Undecided
     Assignee: Abhijeet Malawade (abhijeet-malawade)
         Status: New

** Changed in: glance
     Assignee: (unassigned) => Abhijeet Malawade (abhijeet-malawade)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1401778

Title:
  500 error returned while uploading image using multi filesystem store
  and 'filesystem_store_metadata_file' option enabled

Status in OpenStack Image Registry and Delivery Service (Glance):
  New

Bug description:
  When we try to upload image with multi filesystem store enabled and 'filesystem_store_metadata_file' 
  (containing list of all directories configured for multi filesystem store) is provided
  then glance throws 'HTTPInternalServerError (HTTP 500)' error.

  - Glance Configuration:

  1. /etc/glance/glance-api.conf

  [DEFAULT]
  show_multiple_locations = True
  filesystem_store_metadata_file = /etc/glance/metadata.json

  [glance_store]
  filesystem_store_datadirs = /var/lib/glance/images1/:1
  filesystem_store_datadirs = /var/lib/glance/images2/:2

  
  2. /etc/glance/metadata.json

  [
      {
          "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
          "mountpoint": "/var/lib/glance/images1/" 
      },
      {
          "id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2",
          "mountpoint": "/var/lib/glance/images2/" 
      }
  ]

  3. 'df -ha' command result:

  openstack@openstack:/opt/stack$ df -ha
  Filesystem                           Size  Used Avail Use% Mounted on
  proc                                    0     0     0    - /proc
  sysfs                                   0     0     0    - /sys
  tmpfs                                799M  600K  798M   1% /run
  /dev/sda1                            236M   41M  183M  19% /boot
  nfsd                                    0     0     0    - /proc/fs/nfsd
  10.69.4.173:/export/images2          443G   23G  398G   6% /var/lib/glance/images2
  10.69.4.172:/export/images1          447G  8.7G  415G   3% /var/lib/glance/images1
  openstack@openstack:/opt/stack$

  
  - Steps to reproduce:

  1. Create image:
  glance --os-image-api-version 2 image-create --name Test123 --disk-format raw --container-format ami

  2. Upload image data:
   
  openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload 47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py

  HTTPInternalServerError (HTTP 500)
  openstack@openstack-150:~$

  
  - glance-api.log :

  2014-12-11 22:16:59.586 3495 ERROR glance_store.backend [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver <glance_store._drivers.filesystem.Store
   object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-
  8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
  2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to internal error
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most recent call last):
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data image.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.base.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 252, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.notifier.error('image.upload', msg)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 201, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 176, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 364, in set_data
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data context=self.context)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in add_to_backend
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in store_add_to_backend
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise exceptions.BackendException(msg)
  2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException: The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type

  
  When we enable 'filesystem_store_metadata_file' file option in glance-api.conf then it adds this json data as metadata for all glance images.
  This data is then used by NOVA to directly copy image using 'cp' command. (when glance image directories are shared between nova-compute and glance)
  For direct download in nova.conf we need to enable 'allowed_direct_url_schemes' option.
  Also nova needs 'id' and 'moutpoint' keys to be present in glance-image metadata for direct image copy.

  If there is only one glance image directory (ie- one nfs mountpoint) then we do not get this '500 internal server error'.
  But if there are multiple image data directories configured and in metadata JSON file we give list of dicts containing all 
  mountpoints then we get above error.

  IMO we need to validate glance metadata JSON file (for 'id' and
  'mountpoint' keys) before adding it as image-metadata and if there are
  multiple mountpoints (directories) configured for glance then we need
  to add appropriate metadata to glance-image instead of adding all
  image metadata which is provided in JSON file.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1401778/+subscriptions

Follow ups

[Bug 1401778] Re: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Doug Hellmann, 2015-12-02
[Bug 1401778] Re: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: nikhil komawar, 2015-02-06
[Bug 1401778] Re: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Thierry Carrez, 2015-02-05
[Bug 1401778] Re: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Pranali Deore, 2015-01-07
[Bug 1401778] Re: 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Abhijeet Malawade, 2014-12-12
[Bug 1401778] [NEW] 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Abhijeet Malawade, 2014-12-12

References

[Bug 1401778] [NEW] 500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled
From: Abhijeet Malawade, 2014-12-12