yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #79004
[Bug 1833739] [NEW] keystone (stein), python3, and postgresql: hex in database
Public bug reported:
When using Keystone (Stein) with Python3 and PostgreSQL, and creating
new credentials, we end up with hex in the database for the
encrypted_blob in the credential. This causes Keystone to start throwing
errors, as it cannot decrypt the encrypted_blob. Adding debugging code
to various portions of Keystone narrowed down the location of this
problem.
Relevant information, and examples will follow.
Here is the error we were seeing when we'd create a new credential:
2019-06-21 15:18:10.189 21487 ERROR
keystone.credential.providers.fernet.core [req-
ff10ee5f-c009-487a-8091-90a65b55eb80 STUFF STUFF - STUFF STUFF]
Credential could not be decrypted. Please contact the administrator:
cryptography.fernet.InvalidToken
After doing a lot of tracing, we ended up mutating:
https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L41
@sql.handle_conflicts(conflict_type='credential')
def create_credential(self, credential_id, credential):
testout = ("cred in sql function: ", credential['encrypted_blob'])
LOG.error(testout)
testout = ("cred in sql function type: ", type(credential['encrypted_blob']))
LOG.error(testout)
with sql.session_for_write() as session:
ref = CredentialModel.from_dict(credential)
testout2 = ("cred in sql function model: ", ref['encrypted_blob'])
LOG.error(testout2)
testout3 = ("cred in sql function model type: ", type(ref['encrypted_blob']))
LOG.error(testout3)
session.add(ref)
return ref.to_dict()
Here is what the output looks like when generating a credential:
2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function type: ', <class 'bytes'>)
2019-06-21 15:18:10.091 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.092 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model type: ', <class 'bytes'>)
This is what it looks like in the database:
-[ RECORD 6 ]--+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | IDWASHERE
user_id | USERIDWASHERE
project_id | PROJECTIDWASHERE
type | ec2
extra | {}
key_hash | KEYHASHWASHERE
encrypted_blob | \x67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d
If you take that encrypted_blob, and decode it:
>>> import codecs
>>> codecs.decode("67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d", "hex")
b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs='
>>>
Database and database server info:
Name | keystone
Owner | postgres
Encoding | UTF8
Collate | en_US.UTF-8
Ctype | en_US.UTF-8
ii postgresql-9.5 9.5.17-0ubuntu0.16.04.1 amd64 object-relational SQL database, version 9.5 server
ii postgresql-client-9.5 9.5.17-0ubuntu0.16.04.1 amd64 front-end programs for PostgreSQL 9.5
ii postgresql-client-common 173ubuntu0.2 all manager for multiple PostgreSQL client versions
ii postgresql-common 173ubuntu0.2 all PostgreSQL database-cluster manager
ii postgresql-contrib-9.5 9.5.17-0ubuntu0.16.04.1 amd64 additional facilities for PostgreSQL
postgres@openstack-db01:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
Keystone and associated oslo.db/sqlalchemy/psycopg information:
ii python3-keystone 2:15.0.0-0ubuntu1~cloud0 all OpenStack identity service - Python 3 library
ii python3-keystoneauth1 3.13.1-0ubuntu1~cloud0 all authentication library for OpenStack Identity - Python 3.x
ii python3-keystoneclient 1:3.19.0-0ubuntu1~cloud0 all client library for the OpenStack Keystone API - Python 3.x
ii python3-keystonemiddleware 6.0.0-0ubuntu1~cloud0 all Middleware for OpenStack Identity (Keystone) - Python 3.x
ii python3-oslo.db 4.44.0-0ubuntu1~cloud0
all database connectivity to the different backends and helper
utils - Python 3.x
ii python3-sqlalchemy 1.2.15+ds1-1~cloud0 all SQL toolkit and Object Relational Mapper for Python 3
ii python3-sqlalchemy-ext 1.2.15+ds1-1~cloud0 amd64 SQL toolkit and Object Relational Mapper for Python3 - C extension
ii python3-psycopg2 2.7.4-1
amd64 Python 3 module for PostgreSQL
root@keystone01:/usr/lib/python3/dist-packages# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Using the exact same database server, running the Rocky version of
Keystone with Python2, we do not see this problem. Keystone is being
installed using the Ubuntu Cloud Archive:
https://wiki.ubuntu.com/OpenStack/CloudArchive
** Affects: keystone
Importance: Undecided
Status: New
** Tags: stein-backport-potential
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1833739
Title:
keystone (stein), python3, and postgresql: hex in database
Status in OpenStack Identity (keystone):
New
Bug description:
When using Keystone (Stein) with Python3 and PostgreSQL, and creating
new credentials, we end up with hex in the database for the
encrypted_blob in the credential. This causes Keystone to start
throwing errors, as it cannot decrypt the encrypted_blob. Adding
debugging code to various portions of Keystone narrowed down the
location of this problem.
Relevant information, and examples will follow.
Here is the error we were seeing when we'd create a new credential:
2019-06-21 15:18:10.189 21487 ERROR
keystone.credential.providers.fernet.core [req-
ff10ee5f-c009-487a-8091-90a65b55eb80 STUFF STUFF - STUFF STUFF]
Credential could not be decrypted. Please contact the administrator:
cryptography.fernet.InvalidToken
After doing a lot of tracing, we ended up mutating:
https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L41
@sql.handle_conflicts(conflict_type='credential')
def create_credential(self, credential_id, credential):
testout = ("cred in sql function: ", credential['encrypted_blob'])
LOG.error(testout)
testout = ("cred in sql function type: ", type(credential['encrypted_blob']))
LOG.error(testout)
with sql.session_for_write() as session:
ref = CredentialModel.from_dict(credential)
testout2 = ("cred in sql function model: ", ref['encrypted_blob'])
LOG.error(testout2)
testout3 = ("cred in sql function model type: ", type(ref['encrypted_blob']))
LOG.error(testout3)
session.add(ref)
return ref.to_dict()
Here is what the output looks like when generating a credential:
2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function type: ', <class 'bytes'>)
2019-06-21 15:18:10.091 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.092 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model type: ', <class 'bytes'>)
This is what it looks like in the database:
-[ RECORD 6 ]--+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | IDWASHERE
user_id | USERIDWASHERE
project_id | PROJECTIDWASHERE
type | ec2
extra | {}
key_hash | KEYHASHWASHERE
encrypted_blob | \x67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d
If you take that encrypted_blob, and decode it:
>>> import codecs
>>> codecs.decode("67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d", "hex")
b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs='
>>>
Database and database server info:
Name | keystone
Owner | postgres
Encoding | UTF8
Collate | en_US.UTF-8
Ctype | en_US.UTF-8
ii postgresql-9.5 9.5.17-0ubuntu0.16.04.1 amd64 object-relational SQL database, version 9.5 server
ii postgresql-client-9.5 9.5.17-0ubuntu0.16.04.1 amd64 front-end programs for PostgreSQL 9.5
ii postgresql-client-common 173ubuntu0.2 all manager for multiple PostgreSQL client versions
ii postgresql-common 173ubuntu0.2 all PostgreSQL database-cluster manager
ii postgresql-contrib-9.5 9.5.17-0ubuntu0.16.04.1 amd64 additional facilities for PostgreSQL
postgres@openstack-db01:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
Keystone and associated oslo.db/sqlalchemy/psycopg information:
ii python3-keystone 2:15.0.0-0ubuntu1~cloud0 all OpenStack identity service - Python 3 library
ii python3-keystoneauth1 3.13.1-0ubuntu1~cloud0 all authentication library for OpenStack Identity - Python 3.x
ii python3-keystoneclient 1:3.19.0-0ubuntu1~cloud0 all client library for the OpenStack Keystone API - Python 3.x
ii python3-keystonemiddleware 6.0.0-0ubuntu1~cloud0 all Middleware for OpenStack Identity (Keystone) - Python 3.x
ii python3-oslo.db 4.44.0-0ubuntu1~cloud0
all database connectivity to the different backends and
helper utils - Python 3.x
ii python3-sqlalchemy 1.2.15+ds1-1~cloud0 all SQL toolkit and Object Relational Mapper for Python 3
ii python3-sqlalchemy-ext 1.2.15+ds1-1~cloud0 amd64 SQL toolkit and Object Relational Mapper for Python3 - C extension
ii python3-psycopg2 2.7.4-1
amd64 Python 3 module for PostgreSQL
root@keystone01:/usr/lib/python3/dist-packages# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Using the exact same database server, running the Rocky version of
Keystone with Python2, we do not see this problem. Keystone is being
installed using the Ubuntu Cloud Archive:
https://wiki.ubuntu.com/OpenStack/CloudArchive
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1833739/+subscriptions
Follow ups
