yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1478961] [NEW] db sync on federation failed if there is existing data

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jose Castro Leon <jose.castro.leon@xxxxxxx>
Date: Tue, 28 Jul 2015 13:22:42 -0000
Reply-to: Bug 1478961 <1478961@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If you have an existing entry in the identity_provider table, when
updating from juno to kilo, it fails with the following error

OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot
be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s,
%s)' ('https://cern.ch/login', None)

The migrate_repo goes from 2 to 6 and it failes in step 7.

The issue is linked to a new field created on identity_provider table called 'remote_id' that it is created in step 3 and left empty.
Then on step 7 it tries to read to insert into the idp_remote_ids that does not accept null values.

2015-07-28 15:04:00.247 18168 TRACE keystone Traceback (most recent call last):
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/bin/keystone-manage", line 44, in <module>
2015-07-28 15:04:00.247 18168 TRACE keystone     cli.main(argv=sys.argv, config_files=config_files)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 585, in main
2015-07-28 15:04:00.247 18168 TRACE keystone     CONF.command.cmd_class.main()
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 76, in main
2015-07-28 15:04:00.247 18168 TRACE keystone     migration_helpers.sync_database_to_version(extension, version)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 249, in sync_database_to_version
2015-07-28 15:04:00.247 18168 TRACE keystone     _sync_extension_repo(extension, version)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 219, in _sync_extension_repo
2015-07-28 15:04:00.247 18168 TRACE keystone     init_version=init_version)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/migration.py", line 79, in db_sync
2015-07-28 15:04:00.247 18168 TRACE keystone     return versioning_api.upgrade(engine, repository, version)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
2015-07-28 15:04:00.247 18168 TRACE keystone     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "<string>", line 2, in _migrate
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2015-07-28 15:04:00.247 18168 TRACE keystone     return f(*a, **kw)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
2015-07-28 15:04:00.247 18168 TRACE keystone     schema.runchange(ver, change, changeset.step)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
2015-07-28 15:04:00.247 18168 TRACE keystone     change.run(self.engine, step)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
2015-07-28 15:04:00.247 18168 TRACE keystone     script_func(engine)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/contrib/federation/migrate_repo/versions/007_add_remote_id_table.py", line 39, in upgrade
2015-07-28 15:04:00.247 18168 TRACE keystone     remote_id_table.insert(remote_idp_entry).execute()
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/base.py", line 386, in execute
2015-07-28 15:04:00.247 18168 TRACE keystone     return e._execute_clauseelement(self, multiparams, params)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1759, in _execute_clauseelement
2015-07-28 15:04:00.247 18168 TRACE keystone     return connection._execute_clauseelement(elem, multiparams, params)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
2015-07-28 15:04:00.247 18168 TRACE keystone     compiled_sql, distilled_params
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
2015-07-28 15:04:00.247 18168 TRACE keystone     context)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception
2015-07-28 15:04:00.247 18168 TRACE keystone     e, statement, parameters, cursor, context)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
2015-07-28 15:04:00.247 18168 TRACE keystone     util.raise_from_cause(newraise, exc_info)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-28 15:04:00.247 18168 TRACE keystone     reraise(type(exception), exception, tb=exc_tb)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
2015-07-28 15:04:00.247 18168 TRACE keystone     context)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
2015-07-28 15:04:00.247 18168 TRACE keystone     cursor.execute(statement, parameters)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2015-07-28 15:04:00.247 18168 TRACE keystone     self.errorhandler(self, exc, value)
2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-07-28 15:04:00.247 18168 TRACE keystone     raise errorclass, errorvalue
2015-07-28 15:04:00.247 18168 TRACE keystone OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s, %s)' ('https://cern.ch/login', None)

** Affects: keystone
     Importance: Undecided
     Assignee: Marek Denis (marek-denis)
         Status: New


** Tags: federation sql

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1478961

Title:
  db sync on federation failed if there is existing data

Status in Keystone:
  New

Bug description:
  If you have an existing entry in the identity_provider table, when
  updating from juno to kilo, it fails with the following error

  OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot
  be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s,
  %s)' ('https://cern.ch/login', None)

  The migrate_repo goes from 2 to 6 and it failes in step 7.

  The issue is linked to a new field created on identity_provider table called 'remote_id' that it is created in step 3 and left empty.
  Then on step 7 it tries to read to insert into the idp_remote_ids that does not accept null values.

  2015-07-28 15:04:00.247 18168 TRACE keystone Traceback (most recent call last):
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/bin/keystone-manage", line 44, in <module>
  2015-07-28 15:04:00.247 18168 TRACE keystone     cli.main(argv=sys.argv, config_files=config_files)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 585, in main
  2015-07-28 15:04:00.247 18168 TRACE keystone     CONF.command.cmd_class.main()
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/cli.py", line 76, in main
  2015-07-28 15:04:00.247 18168 TRACE keystone     migration_helpers.sync_database_to_version(extension, version)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 249, in sync_database_to_version
  2015-07-28 15:04:00.247 18168 TRACE keystone     _sync_extension_repo(extension, version)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 219, in _sync_extension_repo
  2015-07-28 15:04:00.247 18168 TRACE keystone     init_version=init_version)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/migration.py", line 79, in db_sync
  2015-07-28 15:04:00.247 18168 TRACE keystone     return versioning_api.upgrade(engine, repository, version)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
  2015-07-28 15:04:00.247 18168 TRACE keystone     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "<string>", line 2, in _migrate
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
  2015-07-28 15:04:00.247 18168 TRACE keystone     return f(*a, **kw)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
  2015-07-28 15:04:00.247 18168 TRACE keystone     schema.runchange(ver, change, changeset.step)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
  2015-07-28 15:04:00.247 18168 TRACE keystone     change.run(self.engine, step)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
  2015-07-28 15:04:00.247 18168 TRACE keystone     script_func(engine)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/contrib/federation/migrate_repo/versions/007_add_remote_id_table.py", line 39, in upgrade
  2015-07-28 15:04:00.247 18168 TRACE keystone     remote_id_table.insert(remote_idp_entry).execute()
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/base.py", line 386, in execute
  2015-07-28 15:04:00.247 18168 TRACE keystone     return e._execute_clauseelement(self, multiparams, params)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1759, in _execute_clauseelement
  2015-07-28 15:04:00.247 18168 TRACE keystone     return connection._execute_clauseelement(elem, multiparams, params)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
  2015-07-28 15:04:00.247 18168 TRACE keystone     compiled_sql, distilled_params
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
  2015-07-28 15:04:00.247 18168 TRACE keystone     context)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception
  2015-07-28 15:04:00.247 18168 TRACE keystone     e, statement, parameters, cursor, context)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
  2015-07-28 15:04:00.247 18168 TRACE keystone     util.raise_from_cause(newraise, exc_info)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2015-07-28 15:04:00.247 18168 TRACE keystone     reraise(type(exception), exception, tb=exc_tb)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
  2015-07-28 15:04:00.247 18168 TRACE keystone     context)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
  2015-07-28 15:04:00.247 18168 TRACE keystone     cursor.execute(statement, parameters)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
  2015-07-28 15:04:00.247 18168 TRACE keystone     self.errorhandler(self, exc, value)
  2015-07-28 15:04:00.247 18168 TRACE keystone   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2015-07-28 15:04:00.247 18168 TRACE keystone     raise errorclass, errorvalue
  2015-07-28 15:04:00.247 18168 TRACE keystone OperationalError: (OperationalError) (1048, "Column 'remote_id' cannot be null") 'INSERT INTO idp_remote_ids (idp_id, remote_id) VALUES (%s, %s)' ('https://cern.ch/login', None)

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1478961/+subscriptions
Follow ups

[Bug 1478961] Re: db sync on federation failed if there is existing data
From: Thierry Carrez, 2015-10-01