← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1370954] [NEW] glance 500's when passed image name with a 4-byte utf-8 character

 

Public bug reported:

Glance currently 500's when passed an image name with a 4-byte utf-8
character in the name. This is because the mysql utf-8 type only handles
up to 3-byte utf-8 characters:

See http://stackoverflow.com/questions/10957238/incorrect-string-value-
when-trying-to-insert-utf-8-into-mysql-via-jdbc

You can replicate this by using the tempest test: 
test_create_image_specify_multibyte_character_image_name (the positive not negative one)
You'll need to change utf8_name to 
utf8_name = data_utils.rand_name('\xF0\x9F\x92\xA9')

(note removal of unicode prefix)


Backtrace from nova request to glance that triggers this:

2014-09-18 07:35:47.343 843 DEBUG routes.middleware [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644
a9ee2a0433f45d0b - - -] Route path: '/images', defaults: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x7f90c
47a6d10>} __call__ /usr/local/lib/python2.7/dist-packages/routes/middleware.py:102
2014-09-18 07:35:47.343 843 DEBUG routes.middleware [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644
a9ee2a0433f45d0b - - -] Match dict: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x7f90c47a6d10>} __call__ /u
sr/local/lib/python2.7/dist-packages/routes/middleware.py:103
2014-09-18 07:35:47.348 843 ERROR glance.registry.api.v1.images [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 3376
0203c2944644a9ee2a0433f45d0b - - -] Unable to create image None
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images Traceback (most recent call last):
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/registry/api/v1/images.py", line 424, in c
reate
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     image_data = self.db_api.image_create(req.context, image_data)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/api.py", line 124, in image_
create
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return _image_update(context, values, None, purge_props=False)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 92, in wr
apped_f
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 239, in c
all
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return attempt.get(self._wrap_exception)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     reraise(self.value[0], self.value[1], self.value[2])
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 233, in call
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/api.py", line 759, in _image_update
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     image_ref.save(session=session)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/models.py", line 77, in save
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     super(GlanceBase, self).save(session or db_api.get_session())
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/models.py", line 48, in save
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     session.flush()
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1818, in flush
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     self._flush(objects)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1936, in _flush
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     transaction.rollback(_capture_exception=True)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     compat.reraise(exc_type, exc_value, exc_tb)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1900, in _flush
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     flush_context.execute()
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     rec.execute(self)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", lin
e 525, in execute
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     uow
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     table, insert)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     execute(statement, params)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     params)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     compiled_sql, distilled_params
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     context)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     six.reraise(type(newraise), newraise, sys.exc_info()[2])
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     per_fn = fn(ctx)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     context.is_disconnect)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 254, in _raise_operational_errors_directly_filter
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     raise operational_error
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images OperationalError: (OperationalError) (1366, "Incorrect string value: '\\xF0\\x9F\\x92\\xA9-1...' for column 'name' at row 1") 'INSERT INTO images (created_at, updated_at, deleted_at, deleted, id, name, disk_format, container_format, size, virtual_size, status, is_public, checksum, min_disk, min_ram, owner, protected) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2014, 9, 18, 7, 35, 47, 347058), datetime.datetime(2014, 9, 18, 7, 35, 47, 347072), None, 0, '5ff754de-4b8c-4cca-818c-d235dda03b83', '\xf0\x9f\x92\xa9-1786046285', 'ami', 'ami', 0, None, 'queued', 0, None, 0, 0, '33760203c2944644a9ee2a0433f45d0b', 0)
2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images
2014-09-18 07:35:47.356 843 INFO glance.wsgi.server [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644a9ee2a0433f45d0b - - -] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 433, in handle_one_response
    result = self.application(self.environ, start_response)

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1370954

Title:
  glance 500's when passed image name with a 4-byte utf-8 character

Status in OpenStack Image Registry and Delivery Service (Glance):
  New

Bug description:
  Glance currently 500's when passed an image name with a 4-byte utf-8
  character in the name. This is because the mysql utf-8 type only
  handles up to 3-byte utf-8 characters:

  See http://stackoverflow.com/questions/10957238/incorrect-string-
  value-when-trying-to-insert-utf-8-into-mysql-via-jdbc

  You can replicate this by using the tempest test: 
  test_create_image_specify_multibyte_character_image_name (the positive not negative one)
  You'll need to change utf8_name to 
  utf8_name = data_utils.rand_name('\xF0\x9F\x92\xA9')

  (note removal of unicode prefix)

  
  Backtrace from nova request to glance that triggers this:

  2014-09-18 07:35:47.343 843 DEBUG routes.middleware [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644
  a9ee2a0433f45d0b - - -] Route path: '/images', defaults: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x7f90c
  47a6d10>} __call__ /usr/local/lib/python2.7/dist-packages/routes/middleware.py:102
  2014-09-18 07:35:47.343 843 DEBUG routes.middleware [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644
  a9ee2a0433f45d0b - - -] Match dict: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x7f90c47a6d10>} __call__ /u
  sr/local/lib/python2.7/dist-packages/routes/middleware.py:103
  2014-09-18 07:35:47.348 843 ERROR glance.registry.api.v1.images [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 3376
  0203c2944644a9ee2a0433f45d0b - - -] Unable to create image None
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images Traceback (most recent call last):
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/registry/api/v1/images.py", line 424, in c
  reate
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     image_data = self.db_api.image_create(req.context, image_data)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/api.py", line 124, in image_
  create
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return _image_update(context, values, None, purge_props=False)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 92, in wr
  apped_f
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return Retrying(*dargs, **dkw).call(f, *args, **kw)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 239, in c
  all
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     return attempt.get(self._wrap_exception)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     reraise(self.value[0], self.value[1], self.value[2])
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 233, in call
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/api.py", line 759, in _image_update
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     image_ref.save(session=session)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/opt/stack/glance/glance/db/sqlalchemy/models.py", line 77, in save
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     super(GlanceBase, self).save(session or db_api.get_session())
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/models.py", line 48, in save
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     session.flush()
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1818, in flush
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     self._flush(objects)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1936, in _flush
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     transaction.rollback(_capture_exception=True)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     compat.reraise(exc_type, exc_value, exc_tb)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1900, in _flush
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     flush_context.execute()
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     rec.execute(self)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", lin
  e 525, in execute
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     uow
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     table, insert)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     execute(statement, params)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     params)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     compiled_sql, distilled_params
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     context)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     six.reraise(type(newraise), newraise, sys.exc_info()[2])
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     per_fn = fn(ctx)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     context.is_disconnect)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 254, in _raise_operational_errors_directly_filter
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images     raise operational_error
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images OperationalError: (OperationalError) (1366, "Incorrect string value: '\\xF0\\x9F\\x92\\xA9-1...' for column 'name' at row 1") 'INSERT INTO images (created_at, updated_at, deleted_at, deleted, id, name, disk_format, container_format, size, virtual_size, status, is_public, checksum, min_disk, min_ram, owner, protected) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2014, 9, 18, 7, 35, 47, 347058), datetime.datetime(2014, 9, 18, 7, 35, 47, 347072), None, 0, '5ff754de-4b8c-4cca-818c-d235dda03b83', '\xf0\x9f\x92\xa9-1786046285', 'ami', 'ami', 0, None, 'queued', 0, None, 0, 0, '33760203c2944644a9ee2a0433f45d0b', 0)
  2014-09-18 07:35:47.348 843 TRACE glance.registry.api.v1.images
  2014-09-18 07:35:47.356 843 INFO glance.wsgi.server [797abe72-9a70-488e-9254-c71888536278 a2bb050fa6e647398991ebc635741cb1 33760203c2944644a9ee2a0433f45d0b - - -] Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 433, in handle_one_response
      result = self.application(self.environ, start_response)

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1370954/+subscriptions


Follow ups

References