yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1568436] Re: Neutron: postgresql db migration alter_enum issue

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1568436@xxxxxxxxxxxxxxxxxx>
Date: Mon, 11 Apr 2016 14:53:40 -0000
Reply-to: Bug 1568436 <1568436@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/303790
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8d6f3bde1b6219e388e30388d1e2788f50e56850
Submitter: Jenkins
Branch:    master

commit 8d6f3bde1b6219e388e30388d1e2788f50e56850
Author: nick.zhuyj <nick.zhuyj@xxxxxxxxxx>
Date:   Sat Apr 9 21:42:52 2016 -0500

    Postgresql: add do_drop arg in alter_enum function
    
    When postgresql is used as the backend and alter_enum is called.
    The old type of enum will be dropped. But there is a case which
    the old type is used by more than one column, in different table
    eg. Then the drop operation will be failed. Add do_drop,do_rename,
    do_create to let developer to decide whether these operations should
    be skipped or not.
    
    Change-Id: I708288e2cc507017d9f6512b567e3bfc77dfd761
    Closes-Bug: #1568436


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1568436

Title:
  Neutron: postgresql db migration alter_enum issue

Status in neutron:
  Fix Released

Bug description:
  when postgresql is used and we want migration enum type fields. The old type of enum will be dropped in alter_enum function.
  But when the old enum type is used by another table, drop will failed, thus db migration fail.

  Solution:
  Add do_drop as one arg of alter_enum, let developer to decide when to drop old type.

  The issue I met is:
  2016-04-09 18:28:54.134 | 2016-04-09 18:28:54.095 | {4} neutron_vpnaas.tests.functional.openswan.test_openswan_driver.TestOpenSwanDeviceDriver.test_status_reporting [0.696702s] ... ok
  2016-04-09 18:28:54.134 | 2016-04-09 18:28:54.097 | {3} neutron_vpnaas.tests.functional.common.test_migrations_sync.TestModelsMigrationsPsql.test_models_sync [4.424209s] ... FAILED
  2016-04-09 18:28:54.134 | 2016-04-09 18:28:54.099 | 
  2016-04-09 18:28:54.134 | 2016-04-09 18:28:54.101 | Captured stdout:
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.102 | ~~~~~~~~~~~~~~~~
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.104 |     Running upgrade for neutron ...
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.106 |     OK
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.108 |     Running upgrade for neutron-vpnaas ...
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.109 | 
  2016-04-09 18:28:54.135 | 2016-04-09 18:28:54.111 | Captured traceback:
  2016-04-09 18:28:54.136 | 2016-04-09 18:28:54.113 | ~~~~~~~~~~~~~~~~~~~
  2016-04-09 18:28:54.136 | 2016-04-09 18:28:54.115 |     Traceback (most recent call last):
  2016-04-09 18:28:54.137 | 2016-04-09 18:28:54.117 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/test_migrations.py", line 599, in test_models_sync
  2016-04-09 18:28:54.139 | 2016-04-09 18:28:54.119 |         self.db_sync(self.get_engine())
  2016-04-09 18:28:54.141 | 2016-04-09 18:28:54.121 |       File "neutron_vpnaas/tests/functional/common/test_migrations_sync.py", line 35, in db_sync
  2016-04-09 18:28:54.143 | 2016-04-09 18:28:54.122 |         migration.do_alembic_command(conf, 'upgrade', 'heads')
  2016-04-09 18:28:54.145 | 2016-04-09 18:28:54.124 |       File "/opt/stack/new/neutron/neutron/db/migration/cli.py", line 131, in do_alembic_command
  2016-04-09 18:28:54.176 | 2016-04-09 18:28:54.126 |         getattr(alembic_command, cmd)(config, *args, **kwargs)
  2016-04-09 18:28:54.176 | 2016-04-09 18:28:54.128 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
  2016-04-09 18:28:54.176 | 2016-04-09 18:28:54.130 |         script.run_env()
  2016-04-09 18:28:54.176 | 2016-04-09 18:28:54.132 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/script/base.py", line 397, in run_env
  2016-04-09 18:28:54.177 | 2016-04-09 18:28:54.134 |         util.load_python_file(self.dir, 'env.py')
  2016-04-09 18:28:54.177 | 2016-04-09 18:28:54.135 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
  2016-04-09 18:28:54.177 | 2016-04-09 18:28:54.137 |         module = load_module_py(module_id, path)
  2016-04-09 18:28:54.177 | 2016-04-09 18:28:54.139 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
  2016-04-09 18:28:54.177 | 2016-04-09 18:28:54.141 |         mod = imp.load_source(module_id, path, fp)
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.143 |       File "neutron_vpnaas/db/migration/alembic_migrations/env.py", line 87, in <module>
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.145 |         run_migrations_online()
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.147 |       File "neutron_vpnaas/db/migration/alembic_migrations/env.py", line 78, in run_migrations_online
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.149 |         context.run_migrations()
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.151 |       File "<string>", line 8, in run_migrations
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.152 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
  2016-04-09 18:28:54.178 | 2016-04-09 18:28:54.154 |         self.get_context().run_migrations(**kw)
  2016-04-09 18:28:54.179 | 2016-04-09 18:28:54.156 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations
  2016-04-09 18:28:54.179 | 2016-04-09 18:28:54.158 |         step.migration_fn(**kw)
  2016-04-09 18:28:54.180 | 2016-04-09 18:28:54.160 |       File "/opt/stack/new/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/versions/newton/expand/fe637dc3f042_support_sha256.py", line 41, in upgrade
  2016-04-09 18:28:54.182 | 2016-04-09 18:28:54.161 |         nullable=False)
  2016-04-09 18:28:54.184 | 2016-04-09 18:28:54.163 |       File "/opt/stack/new/neutron/neutron/db/migration/__init__.py", line 132, in alter_enum
  2016-04-09 18:28:54.185 | 2016-04-09 18:28:54.165 |         op.execute("DROP TYPE old_%(name)s" % values)
  2016-04-09 18:28:54.187 | 2016-04-09 18:28:54.167 |       File "<string>", line 8, in execute
  2016-04-09 18:28:54.189 | 2016-04-09 18:28:54.168 |       File "<string>", line 3, in execute
  2016-04-09 18:28:54.191 | 2016-04-09 18:28:54.170 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1826, in execute
  2016-04-09 18:28:54.192 | 2016-04-09 18:28:54.172 |         return operations.invoke(op)
  2016-04-09 18:28:54.194 | 2016-04-09 18:28:54.174 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
  2016-04-09 18:28:54.196 | 2016-04-09 18:28:54.175 |         return fn(self, operation)
  2016-04-09 18:28:54.198 | 2016-04-09 18:28:54.177 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 161, in execute_sql
  2016-04-09 18:28:54.220 | 2016-04-09 18:28:54.179 |         execution_options=operation.execution_options
  2016-04-09 18:28:54.220 | 2016-04-09 18:28:54.181 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 121, in execute
  2016-04-09 18:28:54.220 | 2016-04-09 18:28:54.183 |         self._exec(sql, execution_options)
  2016-04-09 18:28:54.220 | 2016-04-09 18:28:54.184 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.186 |         return conn.execute(construct, *multiparams, **params)
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.188 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.190 |         return meth(self, multiparams, params)
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.191 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.193 |         return connection._execute_clauseelement(self, multiparams, params)
  2016-04-09 18:28:54.221 | 2016-04-09 18:28:54.195 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-04-09 18:28:54.222 | 2016-04-09 18:28:54.197 |         compiled_sql, distilled_params
  2016-04-09 18:28:54.222 | 2016-04-09 18:28:54.198 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-04-09 18:28:54.222 | 2016-04-09 18:28:54.200 |         context)
  2016-04-09 18:28:54.222 | 2016-04-09 18:28:54.202 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-04-09 18:28:54.224 | 2016-04-09 18:28:54.204 |         util.raise_from_cause(newraise, exc_info)
  2016-04-09 18:28:54.226 | 2016-04-09 18:28:54.206 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2016-04-09 18:28:54.228 | 2016-04-09 18:28:54.207 |         reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2016-04-09 18:28:54.230 | 2016-04-09 18:28:54.209 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2016-04-09 18:28:54.232 | 2016-04-09 18:28:54.211 |         context)
  2016-04-09 18:28:54.234 | 2016-04-09 18:28:54.213 |       File "/opt/stack/new/neutron-vpnaas/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2016-04-09 18:28:54.235 | 2016-04-09 18:28:54.215 |         cursor.execute(statement, parameters)
  2016-04-09 18:28:54.237 | 2016-04-09 18:28:54.217 |     oslo_db.exception.DBError: (psycopg2.InternalError) cannot drop type old_vpn_auth_algorithms because other objects depend on it
  2016-04-09 18:28:54.239 | 2016-04-09 18:28:54.219 |     DETAIL:  table ipsecpolicies column auth_algorithm depends on type old_vpn_auth_algorithms
  2016-04-09 18:28:54.241 | 2016-04-09 18:28:54.221 |     HINT:  Use DROP ... CASCADE to drop the dependent objects too.
  2016-04-09 18:28:54.243 | 2016-04-09 18:28:54.222 |      [SQL: 'DROP TYPE old_vpn_auth_algorithms']
  2016-04-09 18:28:54.245 | 2016-04-09 18:28:54.224 |     
  2016-04-09 18:28:54.247 | 2016-04-09 18:28:54.226 |

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1568436/+subscriptions
References

[Bug 1568436] [NEW] Neutron: postgresql db migration alter_enum issue
From: Yi Jing Zhu, 2016-04-10