yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1538227] [NEW] Failed `nova-manage db sync` returns exitcode of 0

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Florian Ermisch <0xf10e@xxxxxxxx>
Date: Tue, 26 Jan 2016 17:43:40 -0000
Reply-to: Bug 1538227 <1538227@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We're trying to upgrade to liberty using SaltStack 
(just for context, same issue running the command 
from a shell).

At one point `nova-manage db sync` is executed.
Because of #1511466 we get a *critical* error but
the command still returns an exitcode of 0.
In a somewhat POSIX environment this means
"everything is fine" so the deployment just
continues with an outdated database schema.

          ID: nova-manage db sync
    Function: cmd.run
        Name: nova-manage db sync; sleep 15
      Result: True
     Comment: Command "nova-manage db sync; sleep 15" run
     Started: 18:07:09.511897
    Duration: 17747.739 ms
     Changes:   
              ----------
              pid:
                  18000
              retcode:
                  0
              stderr:
                  No handlers could be found for logger "oslo_config.cfg"
                  2016-01-26 18:07:12.092 18001 DEBUG migrate.versioning.repository [-] Loading repository /usr/lib/python2.7/dist-packages/nova/db/sql
alchemy/migrate_repo... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:76
                  2016-01-26 18:07:12.093 18001 DEBUG migrate.versioning.script.base [-] Loading script /usr/lib/python2.7/dist-packages/nova/db/sqlalc
hemy/migrate_repo/versions/216_havana.py... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27
[...]
[...]
[...]
                  2016-01-26 18:07:12.157 18001 INFO migrate.versioning.api [-] 290 -> 291... 
                  2016-01-26 18:07:12.167 18001 CRITICAL nova [-] ValidationError: There are still 3 unmigrated flavor records. Migration cannot conti$
ue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                  2016-01-26 18:07:12.167 18001 ERROR nova Traceback (most recent call last):
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/bin/nova-manage", line 10, in <module>
                  2016-01-26 18:07:12.167 18001 ERROR nova     sys.exit(main())
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
                  2016-01-26 18:07:12.167 18001 ERROR nova     ret = fn(*fn_args, **fn_kwargs)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
                  2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                  2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_$
ync
                  2016-01-26 18:07:12.167 18001 ERROR nova     version)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
with_engine
                  2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                  2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                  2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_s
ync
                  2016-01-26 18:07:12.167 18001 ERROR nova     version)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
with_engine
                  2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova     schema.runchange(ver, change, changeset.step)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, in runchan$
e
                  2016-01-26 18:07:12.167 18001 ERROR nova     change.run(self.engine, step)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 148, in run
                  2016-01-26 18:07:12.167 18001 ERROR nova     script_func(engine)
                  2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo/versions/291_enfor$
e_flavors_migrated.py", line 35, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova     raise exception.ValidationError(detail=msg)
                  2016-01-26 18:07:12.167 18001 ERROR nova ValidationError: There are still 3 unmigrated flavor records. Migration cannot continue unt$
l all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                  2016-01-26 18:07:12.167 18001 ERROR nova
              stdout:
                  Command failed, please check log for more info
----------
 

root@controller:~# dpkg -l | grep nova
ii  nova-api                            2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - API frontend
ii  nova-cert                           2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - certificate management
ii  nova-common                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
ii  nova-conductor                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - conductor service
ii  nova-consoleauth                    2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - virtual machine scheduler
ii  nova-spiceproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - spice html5 proxy
ii  python-nova                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
ii  python-novaclient                   2:2.30.1-1~cloud0                     all          client library for OpenStack Compute API
root@controller:~# 


How to reproduce:
* setup openstack < liberty
* update pkgs to liberty 
* run `nova-manage db sync && echo "you shouldn't see this because the prev cmd failed" `
(The error while running the command is topic of issue #1511466)

Expected result:
Failing `nova-manage db sync` returns a none-zero exitcode and thus signals a failure
as any utility on a unix-like system should.

Actual result:
Failing `nova-manage db sync` still returns an exitcode of 0 signaling successful
update of database schemas to the running shell/command/automation tool of choice.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1538227

Title:
  Failed `nova-manage db sync` returns exitcode of 0

Status in OpenStack Compute (nova):
  New

Bug description:
  We're trying to upgrade to liberty using SaltStack 
  (just for context, same issue running the command 
  from a shell).

  At one point `nova-manage db sync` is executed.
  Because of #1511466 we get a *critical* error but
  the command still returns an exitcode of 0.
  In a somewhat POSIX environment this means
  "everything is fine" so the deployment just
  continues with an outdated database schema.

            ID: nova-manage db sync
      Function: cmd.run
          Name: nova-manage db sync; sleep 15
        Result: True
       Comment: Command "nova-manage db sync; sleep 15" run
       Started: 18:07:09.511897
      Duration: 17747.739 ms
       Changes:   
                ----------
                pid:
                    18000
                retcode:
                    0
                stderr:
                    No handlers could be found for logger "oslo_config.cfg"
                    2016-01-26 18:07:12.092 18001 DEBUG migrate.versioning.repository [-] Loading repository /usr/lib/python2.7/dist-packages/nova/db/sql
  alchemy/migrate_repo... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:76
                    2016-01-26 18:07:12.093 18001 DEBUG migrate.versioning.script.base [-] Loading script /usr/lib/python2.7/dist-packages/nova/db/sqlalc
  hemy/migrate_repo/versions/216_havana.py... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27
  [...]
  [...]
  [...]
                    2016-01-26 18:07:12.157 18001 INFO migrate.versioning.api [-] 290 -> 291... 
                    2016-01-26 18:07:12.167 18001 CRITICAL nova [-] ValidationError: There are still 3 unmigrated flavor records. Migration cannot conti$
  ue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                    2016-01-26 18:07:12.167 18001 ERROR nova Traceback (most recent call last):
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/bin/nova-manage", line 10, in <module>
                    2016-01-26 18:07:12.167 18001 ERROR nova     sys.exit(main())
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
                    2016-01-26 18:07:12.167 18001 ERROR nova     ret = fn(*fn_args, **fn_kwargs)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_$
  ync
                    2016-01-26 18:07:12.167 18001 ERROR nova     version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
  with_engine
                    2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                    2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_s
  ync
                    2016-01-26 18:07:12.167 18001 ERROR nova     version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
  with_engine
                    2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova     schema.runchange(ver, change, changeset.step)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, in runchan$
  e
                    2016-01-26 18:07:12.167 18001 ERROR nova     change.run(self.engine, step)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 148, in run
                    2016-01-26 18:07:12.167 18001 ERROR nova     script_func(engine)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo/versions/291_enfor$
  e_flavors_migrated.py", line 35, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     raise exception.ValidationError(detail=msg)
                    2016-01-26 18:07:12.167 18001 ERROR nova ValidationError: There are still 3 unmigrated flavor records. Migration cannot continue unt$
  l all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                    2016-01-26 18:07:12.167 18001 ERROR nova
                stdout:
                    Command failed, please check log for more info
  ----------
   

  root@controller:~# dpkg -l | grep nova
  ii  nova-api                            2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - API frontend
  ii  nova-cert                           2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - certificate management
  ii  nova-common                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
  ii  nova-conductor                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - conductor service
  ii  nova-consoleauth                    2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - virtual machine scheduler
  ii  nova-spiceproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - spice html5 proxy
  ii  python-nova                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:2.30.1-1~cloud0                     all          client library for OpenStack Compute API
  root@controller:~# 

  
  How to reproduce:
  * setup openstack < liberty
  * update pkgs to liberty 
  * run `nova-manage db sync && echo "you shouldn't see this because the prev cmd failed" `
  (The error while running the command is topic of issue #1511466)

  Expected result:
  Failing `nova-manage db sync` returns a none-zero exitcode and thus signals a failure
  as any utility on a unix-like system should.

  Actual result:
  Failing `nova-manage db sync` still returns an exitcode of 0 signaling successful
  update of database schemas to the running shell/command/automation tool of choice.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1538227/+subscriptions
Follow ups

[Bug 1538227] Re: Failed `nova-manage db sync` returns exitcode of 0
From: OpenStack Infra, 2016-03-08