yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <2008716@xxxxxxxxxxxxxxxxxx>]

[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2008716

Title:
  nova api_db sync fails remove unused build_requests columns

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Description
  ===========
  During an upgrade from xena to yoga, I followed upgrade steps using openstack-ansible to upgrade nova. During this time i was met with the following log message:

  alembic.runtime.migration [-] Running upgrade d67eeaabee36 ->
  b30f573d3377, Remove unused build_requests columns

  And the python exception:

  /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync
  Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
  An error has occurred:
  Traceback (most recent call last):
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      self.dialect.do_execute(
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      cursor.execute(statement, parameters)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
      result = self._query(query)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
      conn.query(q)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 548, in query
      self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 775, in _read_query_result
      result.read()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 1156, in read
      first_packet = self.connection._read_packet()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 725, in _read_packet
      packet.raise_for_error()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error
      err.raise_mysql_exception(self._data)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
      raise errorclass(errno, errval)
  pymysql.err.OperationalError: (1091, "Can't DROP COLUMN `vm_state`; check that it exists")

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/cmd/manage.py", line 3402, in main
      ret = fn(*fn_args, **fn_kwargs)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/cmd/manage.py", line 612, in sync
      return migration.db_sync(version, database='api')
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/migration.py", line 154, in db_sync
      _upgrade_alembic(engine, config, version)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/migration.py", line 109, in _upgrade_alembic
      alembic_api.upgrade(config, version or 'head')
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
      script.run_env()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
      util.load_python_file(self.dir, "env.py")
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
      module = load_module_py(module_id, path)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
      spec.loader.exec_module(module)  # type: ignore
    File "<frozen importlib._bootstrap_external>", line 848, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/env.py", line 117, in <module>
      run_migrations_online()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/env.py", line 111, in run_migrations_online
      context.run_migrations()
    File "<string>", line 8, in run_migrations
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
      self.get_context().run_migrations(**kw)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
      step.migration_fn(**kw)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/nova/db/api/migrations/versions/b30f573d3377_remove_unused_build_requests_columns.py", line 45, in upgrade
      batch_op.drop_column('request_spec_id')
    File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
      next(self.gen)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
      impl.flush()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/operations/batch.py", line 108, in flush
      fn(*arg, **kw)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/ddl/impl.py", line 329, in drop_column
      self._exec(base.DropColumn(table_name, column, schema=schema))
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
      return conn.execute(construct, multiparams)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
      return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
      return connection._execute_ddl(
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1381, in _execute_ddl
      ret = self._execute_context(
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
      self._handle_dbapi_exception(
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      raise exception
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      self.dialect.do_execute(
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      cursor.execute(statement, parameters)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
      result = self._query(query)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
      conn.query(q)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 548, in query
      self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 775, in _read_query_result
      result.read()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 1156, in read
      first_packet = self.connection._read_packet()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/connections.py", line 725, in _read_packet
      packet.raise_for_error()
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error
      err.raise_mysql_exception(self._data)
    File "/openstack/venvs/nova-25.2.1.dev3/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
      raise errorclass(errno, errval)
  sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1091, "Can't DROP COLUMN `vm_state`; check that it exists")
  [SQL: ALTER TABLE build_requests DROP COLUMN vm_state]

  Steps to reproduce
  ==================
  I was able to reproduce this bug by dropping the entire nova database, creating a nova database and running /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync

  Expected result
  ===============
  I expect nova-manage to test if build_requests is present before attempting to alter/drop it.

  Actual result
  =============
  The python module reports the exception mentioned above.

  Environment
  ===============
  Nova: 25.0.2
  Openstack: Yoga
  Mariadb: 10.6.11
  Hypervisor: libvert+kvm
  Storage Type: ceph
  Network: Neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2008716/+subscriptions