← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1538227] Re: Failed `nova-manage db sync` returns exitcode of 0

 

Reviewed:  https://review.openstack.org/289308
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=173068d52449eb04e905b29da6e417ae69cd8055
Submitter: Jenkins
Branch:    master

commit 173068d52449eb04e905b29da6e417ae69cd8055
Author: Stephen Finucane <stephen.finucane@xxxxxxxxx>
Date:   Mon Mar 7 12:34:47 2016 +0000

    nova-manage: Print, not raise, exceptions
    
    Raising an exception means it is not possible to return a status code.
    Seeing as the 'nova-manage' application is an interactive one, it's
    viable to fix this by printing the exception rather than raising.
    
    Change-Id: Ifa3f80e4f7dccada439ffc9363e9f1504c8c2da1
    Closes-bug: #1538227


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1538227

Title:
  Failed `nova-manage db sync` returns exitcode of 0

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  We're trying to upgrade to liberty using SaltStack 
  (just for context, same issue running the command 
  from a shell).

  At one point `nova-manage db sync` is executed.
  Because of #1511466 we get a *critical* error but
  the command still returns an exitcode of 0.
  In a somewhat POSIX environment this means
  "everything is fine" so the deployment just
  continues with an outdated database schema.

            ID: nova-manage db sync
      Function: cmd.run
          Name: nova-manage db sync; sleep 15
        Result: True
       Comment: Command "nova-manage db sync; sleep 15" run
       Started: 18:07:09.511897
      Duration: 17747.739 ms
       Changes:   
                ----------
                pid:
                    18000
                retcode:
                    0
                stderr:
                    No handlers could be found for logger "oslo_config.cfg"
                    2016-01-26 18:07:12.092 18001 DEBUG migrate.versioning.repository [-] Loading repository /usr/lib/python2.7/dist-packages/nova/db/sql
  alchemy/migrate_repo... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:76
                    2016-01-26 18:07:12.093 18001 DEBUG migrate.versioning.script.base [-] Loading script /usr/lib/python2.7/dist-packages/nova/db/sqlalc
  hemy/migrate_repo/versions/216_havana.py... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27
  [...]
  [...]
  [...]
                    2016-01-26 18:07:12.157 18001 INFO migrate.versioning.api [-] 290 -> 291... 
                    2016-01-26 18:07:12.167 18001 CRITICAL nova [-] ValidationError: There are still 3 unmigrated flavor records. Migration cannot conti$
  ue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                    2016-01-26 18:07:12.167 18001 ERROR nova Traceback (most recent call last):
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/bin/nova-manage", line 10, in <module>
                    2016-01-26 18:07:12.167 18001 ERROR nova     sys.exit(main())
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
                    2016-01-26 18:07:12.167 18001 ERROR nova     ret = fn(*fn_args, **fn_kwargs)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_$
  ync
                    2016-01-26 18:07:12.167 18001 ERROR nova     version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
  with_engine
                    2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                    2016-01-26 18:07:12.167 18001 ERROR nova     return migration.db_sync(version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                    2016-01-26 18:07:12.167 18001 ERROR nova     return IMPL.db_sync(version=version, database=database)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_s
  ync
                    2016-01-26 18:07:12.167 18001 ERROR nova     version)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "<string>", line 2, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in 
  with_engine
                    2016-01-26 18:07:12.167 18001 ERROR nova     return f(*a, **kw)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
                    2016-01-26 18:07:12.167 18001 ERROR nova     schema.runchange(ver, change, changeset.step)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, in runchan$
  e
                    2016-01-26 18:07:12.167 18001 ERROR nova     change.run(self.engine, step)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 148, in run
                    2016-01-26 18:07:12.167 18001 ERROR nova     script_func(engine)
                    2016-01-26 18:07:12.167 18001 ERROR nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo/versions/291_enfor$
  e_flavors_migrated.py", line 35, in upgrade
                    2016-01-26 18:07:12.167 18001 ERROR nova     raise exception.ValidationError(detail=msg)
                    2016-01-26 18:07:12.167 18001 ERROR nova ValidationError: There are still 3 unmigrated flavor records. Migration cannot continue unt$
  l all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                    2016-01-26 18:07:12.167 18001 ERROR nova
                stdout:
                    Command failed, please check log for more info
  ----------
   

  root@controller:~# dpkg -l | grep nova
  ii  nova-api                            2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - API frontend
  ii  nova-cert                           2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - certificate management
  ii  nova-common                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - common files
  ii  nova-conductor                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - conductor service
  ii  nova-consoleauth                    2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                      2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - virtual machine scheduler
  ii  nova-spiceproxy                     2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute - spice html5 proxy
  ii  python-nova                         2:12.0.0-0ubuntu2~cloud0              all          OpenStack Compute Python libraries
  ii  python-novaclient                   2:2.30.1-1~cloud0                     all          client library for OpenStack Compute API
  root@controller:~# 

  
  How to reproduce:
  * setup openstack < liberty
  * update pkgs to liberty 
  * run `nova-manage db sync && echo "you shouldn't see this because the prev cmd failed" `
  (The error while running the command is topic of issue #1511466)

  Expected result:
  Failing `nova-manage db sync` returns a none-zero exitcode and thus signals a failure
  as any utility on a unix-like system should.

  Actual result:
  Failing `nova-manage db sync` still returns an exitcode of 0 signaling successful
  update of database schemas to the running shell/command/automation tool of choice.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1538227/+subscriptions


References