yahoo-eng-team team mailing list archive

[Alexander Arnoldsson <1664207@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Hello. I'm sorry it this is the wrong place to post this; I'm pretty
sure I've messed up myself, but since the error log said "Please report
this", that's what I'm doing.

I've tried to upgrade Openstack from Liberty to Mitaka. It seems the
upgrade broke Nova, it can no longer start.

The upgrade, and nova startup, is handled by Ansible; investigating the nova log files on the control nodes, I found this: 2017-02-13 11:00:00.224 17847 ERROR nova.api.openstack.extensions OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'pci_devices.parent_addr' in 'field list'") ...
2017-02-13 11:00:00.224 17847 ERROR nova.api.openstack.extensions
2017-02-13 11:00:00.226 17847 INFO nova.api.openstack.wsgi [req-14c7b025-0e2c-45e0-b35e-f691eee8ef75 685fd4e2b2f0467c9cec6fca59425d13 3cfd68d01988458db3a8b915975693ff] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>


DEBUG oslo_db.sqlalchemy.engines [req-7e385581-96b3-40f7-a9b4-f2409790ad1d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /opt/stack/venv/nova-20161014T013009Z/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:256
2017-02-13 11:26:21.255 47245 CRITICAL nova [req-7e385581-96b3-40f7-a9b4-f2409790ad1d None None] OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'aggregates.uuid' in 'field list'") ...

I assume that, somehow, the database is not the same version as the nova
code. Feeling fearless, I tried to do a 'db sync', that said the db was
been updated to 319. But after that, a ' select * from
nova.migrate_version;' shows version=302, with an incorrect repository
path (/opt/stack/venv/nova-20160503T082438Z/lib/python2.7/site-
packages/nova/db/sqlalchemy/migrate_repo. expected: ...nova-
20161014T013009Z....

If anyone could provide me with some hints on how to fix the nova upgrade, I'd be as happy as my little pony.
It not, please accept this bug report for the "Unexpected API Error" error message.

Best regards,
Alexander

####################### outputs ####################
mysql> select * from nova.migrate_version;
+---------------+---------------------------------------------------------------------------------------------------+---------+
| repository_id | repository_path                                                                                   | version |
+---------------+---------------------------------------------------------------------------------------------------+---------+
| nova          | /opt/stack/venv/nova-20160503T082438Z/lib/python2.7/site-packages/nova/db/sqlalchemy/migrate_repo |     302 |
+---------------+---------------------------------------------------------------------------------------------------+---------+
#################
/opt/stack/venv/nova-20161014T013009Z/bin/nova-manage version
13.1.3
/opt/stack/venv/nova-20161014T013009Z/bin/nova-scheduler --version
13.1.3
nova --version
3.3.3
sudo mysql --version
mysql  Ver 14.14 Distrib 5.6.30-76.3, for debian-linux-gnu (x86_64) using  6.3

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1664207

Title:
  nova startup failure after openstack upgrade

Status in OpenStack Compute (nova):
  New

Bug description:
  Hello. I'm sorry it this is the wrong place to post this; I'm pretty
  sure I've messed up myself, but since the error log said "Please
  report this", that's what I'm doing.

  I've tried to upgrade Openstack from Liberty to Mitaka. It seems the
  upgrade broke Nova, it can no longer start.

  The upgrade, and nova startup, is handled by Ansible; investigating the nova log files on the control nodes, I found this: 2017-02-13 11:00:00.224 17847 ERROR nova.api.openstack.extensions OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'pci_devices.parent_addr' in 'field list'") ...
  2017-02-13 11:00:00.224 17847 ERROR nova.api.openstack.extensions
  2017-02-13 11:00:00.226 17847 INFO nova.api.openstack.wsgi [req-14c7b025-0e2c-45e0-b35e-f691eee8ef75 685fd4e2b2f0467c9cec6fca59425d13 3cfd68d01988458db3a8b915975693ff] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>

  
  DEBUG oslo_db.sqlalchemy.engines [req-7e385581-96b3-40f7-a9b4-f2409790ad1d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /opt/stack/venv/nova-20161014T013009Z/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:256
  2017-02-13 11:26:21.255 47245 CRITICAL nova [req-7e385581-96b3-40f7-a9b4-f2409790ad1d None None] OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'aggregates.uuid' in 'field list'") ...

  I assume that, somehow, the database is not the same version as the
  nova code. Feeling fearless, I tried to do a 'db sync', that said the
  db was been updated to 319. But after that, a ' select * from
  nova.migrate_version;' shows version=302, with an incorrect repository
  path (/opt/stack/venv/nova-20160503T082438Z/lib/python2.7/site-
  packages/nova/db/sqlalchemy/migrate_repo. expected: ...nova-
  20161014T013009Z....

  If anyone could provide me with some hints on how to fix the nova upgrade, I'd be as happy as my little pony.
  It not, please accept this bug report for the "Unexpected API Error" error message.

  Best regards,
  Alexander

  ####################### outputs ####################
  mysql> select * from nova.migrate_version;
  +---------------+---------------------------------------------------------------------------------------------------+---------+
  | repository_id | repository_path                                                                                   | version |
  +---------------+---------------------------------------------------------------------------------------------------+---------+
  | nova          | /opt/stack/venv/nova-20160503T082438Z/lib/python2.7/site-packages/nova/db/sqlalchemy/migrate_repo |     302 |
  +---------------+---------------------------------------------------------------------------------------------------+---------+
  #################
  /opt/stack/venv/nova-20161014T013009Z/bin/nova-manage version
  13.1.3
  /opt/stack/venv/nova-20161014T013009Z/bin/nova-scheduler --version
  13.1.3
  nova --version
  3.3.3
  sudo mysql --version
  mysql  Ver 14.14 Distrib 5.6.30-76.3, for debian-linux-gnu (x86_64) using  6.3

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1664207/+subscriptions