yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1898489] [NEW] conrollres upgrade didnt failed even though galnce upgrade has faild

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gal Beniluz <1898489@xxxxxxxxxxxxxxxxxx>
Date: Sun, 04 Oct 2020 22:49:56 -0000
Reply-to: Bug 1898489 <1898489@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

During an upgrade process from Queens to Rocky we got an issue related to glance
During the controllers upgrade step the creation of the additonal columns to glance image table in the DB got failed with the following error
"+++ glance-manage db_sync", 
        "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:1352: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade", 
        "  expire_on_commit=expire_on_commit, _conf=conf)", 
        "INFO  [alembic.runtime.migration] Context impl MySQLImpl.", 
        "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", 
        "INFO  [alembic.runtime.migration] Running upgrade queens_expand01 -> rocky_expand01, add os_hidden column to images table", 
        "CRITI [glance] Unhandled error", 
        "Traceback (most recent call last):", 
        "  File \"/usr/bin/glance-manage\", line 10, in <module>", 
        "    sys.exit(main())", 
        "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 563, in main", 
        "    return CONF.command.action_fn()", 
        "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 395, in sync", 
        "    self.command_object.sync(CONF.command.version)", 
        "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 165, in sync", 
        "    self.expand(online_migration=False)", 
        "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 222, in expand", 
        "    self._sync(version=expand_head)", 
        "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 180, in _sync", 
        "    alembic_command.upgrade(a_config, version)", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/command.py\", line 254, in upgrade", 
        "    script.run_env()", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/script/base.py\", line 425, in run_env", 
        "    util.load_python_file(self.dir, 'env.py')", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file", 
        "    module = load_module_py(module_id, path)", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/util/compat.py\", line 141, in load_module_py", 
        "    mod = imp.load_source(module_id, path, fp)", 
        "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/env.py\", line 88, in <module>", 
        "    run_migrations_online()", 
        "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/env.py\", line 83, in run_migrations_online", 
        "    context.run_migrations()", 
        "  File \"<string>\", line 8, in run_migrations", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/runtime/environment.py\", line 836, in run_migrations", 
        "    self.get_context().run_migrations(**kw)", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/runtime/migration.py\", line 330, in run_migrations", 
        "    step.migration_fn(**kw)", 
        "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/versions/rocky_expand01_add_os_hidden.py\", line 31, in upgrade", 
        "    op.add_column('images', h_col)", 
        "  File \"<string>\", line 8, in add_column", 
        "  File \"<string>\", line 3, in add_column", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/operations/ops.py\", line 1565, in add_column", 
        "    return operations.invoke(op)", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/operations/base.py\", line 319, in invoke", 
        "    return fn(self, operation)", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py\", line 123, in add_column", 
        "    schema=schema", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/ddl/impl.py\", line 172, in add_column", 
        "    self._exec(base.AddColumn(table_name, column, schema=schema))", 
        "  File \"/usr/lib/python2.7/site-packages/alembic/ddl/impl.py\", line 118, in _exec", 
        "    return conn.execute(construct, *multiparams, **params)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 948, in execute", 
        "    return meth(self, multiparams, params)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py\", line 68, in _execute_on_connection", 
        "    return connection._execute_ddl(self, multiparams, params)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1009, in _execute_ddl", 
        "    compiled", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1200, in _execute_context", 
        "    context)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1409, in _handle_dbapi_exception", 
        "    util.raise_from_cause(newraise, exc_info)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause", 
        "    reraise(type(exception), exception, tb=exc_tb, cause=cause)", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1193, in _execute_context", 
        "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py\", line 507, in do_execute", 
        "    cursor.execute(statement, parameters)", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/cursors.py\", line 170, in execute", 
        "    result = self._query(query)", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/cursors.py\", line 328, in _query", 
        "    conn.query(q)", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 516, in query", 
        "    self._affected_rows = self._read_query_result(unbuffered=unbuffered)", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 727, in _read_query_result", 
        "    result.read()", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 1066, in read", 
        "    first_packet = self.connection._read_packet()", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 656, in _read_packet", 
        "    packet_header = self._read_bytes(4)", 
        "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 702, in _read_bytes", 
        "    CR.CR_SERVER_LOST, \"Lost connection to MySQL server during query\")", 
        "DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'ALTER TABLE images ADD COLUMN os_hidden BOOL NOT NULL DEFAULT false'] (Background on this error at: http://sqlalche.me/e/e3q8)", 

However, despite that error, ansible didnt stopped or failed it just ignored it and finished successfuly.
As a resault glance is completly not functioing the images table has two missing columns, glance commnads are failing because it is looking for thoese two missing columns and we cannot procced with the upgrade.

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1898489

Title:
  conrollres upgrade didnt failed even though galnce upgrade has faild

Status in Glance:
  New

Bug description:
  During an upgrade process from Queens to Rocky we got an issue related to glance
  During the controllers upgrade step the creation of the additonal columns to glance image table in the DB got failed with the following error
  "+++ glance-manage db_sync", 
          "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:1352: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade", 
          "  expire_on_commit=expire_on_commit, _conf=conf)", 
          "INFO  [alembic.runtime.migration] Context impl MySQLImpl.", 
          "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", 
          "INFO  [alembic.runtime.migration] Running upgrade queens_expand01 -> rocky_expand01, add os_hidden column to images table", 
          "CRITI [glance] Unhandled error", 
          "Traceback (most recent call last):", 
          "  File \"/usr/bin/glance-manage\", line 10, in <module>", 
          "    sys.exit(main())", 
          "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 563, in main", 
          "    return CONF.command.action_fn()", 
          "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 395, in sync", 
          "    self.command_object.sync(CONF.command.version)", 
          "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 165, in sync", 
          "    self.expand(online_migration=False)", 
          "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 222, in expand", 
          "    self._sync(version=expand_head)", 
          "  File \"/usr/lib/python2.7/site-packages/glance/cmd/manage.py\", line 180, in _sync", 
          "    alembic_command.upgrade(a_config, version)", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/command.py\", line 254, in upgrade", 
          "    script.run_env()", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/script/base.py\", line 425, in run_env", 
          "    util.load_python_file(self.dir, 'env.py')", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file", 
          "    module = load_module_py(module_id, path)", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/util/compat.py\", line 141, in load_module_py", 
          "    mod = imp.load_source(module_id, path, fp)", 
          "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/env.py\", line 88, in <module>", 
          "    run_migrations_online()", 
          "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/env.py\", line 83, in run_migrations_online", 
          "    context.run_migrations()", 
          "  File \"<string>\", line 8, in run_migrations", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/runtime/environment.py\", line 836, in run_migrations", 
          "    self.get_context().run_migrations(**kw)", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/runtime/migration.py\", line 330, in run_migrations", 
          "    step.migration_fn(**kw)", 
          "  File \"/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/alembic_migrations/versions/rocky_expand01_add_os_hidden.py\", line 31, in upgrade", 
          "    op.add_column('images', h_col)", 
          "  File \"<string>\", line 8, in add_column", 
          "  File \"<string>\", line 3, in add_column", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/operations/ops.py\", line 1565, in add_column", 
          "    return operations.invoke(op)", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/operations/base.py\", line 319, in invoke", 
          "    return fn(self, operation)", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py\", line 123, in add_column", 
          "    schema=schema", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/ddl/impl.py\", line 172, in add_column", 
          "    self._exec(base.AddColumn(table_name, column, schema=schema))", 
          "  File \"/usr/lib/python2.7/site-packages/alembic/ddl/impl.py\", line 118, in _exec", 
          "    return conn.execute(construct, *multiparams, **params)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 948, in execute", 
          "    return meth(self, multiparams, params)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py\", line 68, in _execute_on_connection", 
          "    return connection._execute_ddl(self, multiparams, params)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1009, in _execute_ddl", 
          "    compiled", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1200, in _execute_context", 
          "    context)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1409, in _handle_dbapi_exception", 
          "    util.raise_from_cause(newraise, exc_info)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause", 
          "    reraise(type(exception), exception, tb=exc_tb, cause=cause)", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1193, in _execute_context", 
          "  File \"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py\", line 507, in do_execute", 
          "    cursor.execute(statement, parameters)", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/cursors.py\", line 170, in execute", 
          "    result = self._query(query)", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/cursors.py\", line 328, in _query", 
          "    conn.query(q)", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 516, in query", 
          "    self._affected_rows = self._read_query_result(unbuffered=unbuffered)", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 727, in _read_query_result", 
          "    result.read()", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 1066, in read", 
          "    first_packet = self.connection._read_packet()", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 656, in _read_packet", 
          "    packet_header = self._read_bytes(4)", 
          "  File \"/usr/lib/python2.7/site-packages/pymysql/connections.py\", line 702, in _read_bytes", 
          "    CR.CR_SERVER_LOST, \"Lost connection to MySQL server during query\")", 
          "DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'ALTER TABLE images ADD COLUMN os_hidden BOOL NOT NULL DEFAULT false'] (Background on this error at: http://sqlalche.me/e/e3q8)", 

  However, despite that error, ansible didnt stopped or failed it just ignored it and finished successfuly.
  As a resault glance is completly not functioing the images table has two missing columns, glance commnads are failing because it is looking for thoese two missing columns and we cannot procced with the upgrade.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1898489/+subscriptions
Follow ups

[Bug 1898489] Re: conrollres upgrade didnt failed even though galnce upgrade has faild
From: Abhishek Kekane, 2021-06-17