yahoo-eng-team team mailing list archive

[Clark Boylan <clark.boylan@xxxxxxxxx>]

Public bug reported:

glance-manage throws errors under the python3.5 job because it attempts
to open and read a file with utf8 characters in it, but devstack has
hard set LANG=C.

ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file
/etc/glance/metadefs/compute-trust.json while populating metadata due
to: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in
range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in
position 90: ordinal not in range(128)

This only happens under python3 because python3 open() will refer to
locale.getpreferredencoding() by default if no encoding is explicitly
set. Python2 doesn't have this problem because strings and open operate
on binary not encoded things.

Devstack sets LANG=C at:
https://git.openstack.org/cgit/openstack-dev/devstack/tree/stack.sh#n30

Example job run where this happens:
http://logs.openstack.org/10/367810/41/check/gate-tempest-dsvm-py35-ubuntu-xenial/89634cf/logs/devstacklog.txt.gz#_2017-06-13_14_25_15_262

One thing that makes this tricky is that open() under python2 doesn't
take an encoding while open() under python3 does. Easy enough to handle
this in code but maybe we should try and get six to address this?

Also worth noting that the infra test nodes should have a locale of
C.utf8 or C.UTF-8, but these locales are apparently (not yet) universal.

Considering that devstack wants to enforce and ascii locale the simplest
option here may just be to remove the utf8 characters from the metadata
json files. '®' and '–' are the two characters which can be replaced
with '(R)' and '-'.

** Affects: devstack
     Importance: Undecided
         Status: New

** Affects: glance
     Importance: Undecided
         Status: New

** Also affects: glance
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1697733

Title:
  LANG is explicitly set to C, but some services (like glance) want to
  read files with utf8 characters

Status in devstack:
  New
Status in Glance:
  New

Bug description:
  glance-manage throws errors under the python3.5 job because it
  attempts to open and read a file with utf8 characters in it, but
  devstack has hard set LANG=C.

  ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file
  /etc/glance/metadefs/compute-trust.json while populating metadata due
  to: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not
  in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte
  0xc2 in position 90: ordinal not in range(128)

  This only happens under python3 because python3 open() will refer to
  locale.getpreferredencoding() by default if no encoding is explicitly
  set. Python2 doesn't have this problem because strings and open
  operate on binary not encoded things.

  Devstack sets LANG=C at:
  https://git.openstack.org/cgit/openstack-dev/devstack/tree/stack.sh#n30

  Example job run where this happens:
  http://logs.openstack.org/10/367810/41/check/gate-tempest-dsvm-py35-ubuntu-xenial/89634cf/logs/devstacklog.txt.gz#_2017-06-13_14_25_15_262

  One thing that makes this tricky is that open() under python2 doesn't
  take an encoding while open() under python3 does. Easy enough to
  handle this in code but maybe we should try and get six to address
  this?

  Also worth noting that the infra test nodes should have a locale of
  C.utf8 or C.UTF-8, but these locales are apparently (not yet)
  universal.

  Considering that devstack wants to enforce and ascii locale the
  simplest option here may just be to remove the utf8 characters from
  the metadata json files. '®' and '–' are the two characters which can
  be replaced with '(R)' and '-'.

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1697733/+subscriptions