yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1824435] [NEW] fill_virtual_interface_list migration fails on second attempt

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: iain MacDonnell <1824435@xxxxxxxxxxxxxxxxxx>
Date: Thu, 11 Apr 2019 22:18:10 -0000
Reply-to: Bug 1824435 <1824435@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On attempting to run online_data_migrations on Stein for the second time
(and beyond), fill_virtual_interface_list fails as below. I find two
rows in the security_groups table which have name='default' and
project_id NULL.

2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage [req-6cb533e2-58b5-41db-a455-29dae8efef31 - - - - -] Error attempting to run <function fill_virtual_interface_list at 0x7f1ed1b6f500>: TypeError: 'NoneType' object has no attribute '__getitem__'
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage Traceback (most recent call last):
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 686, in _run_migration
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     found, done = migration_meth(ctxt, count)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return fn(*args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     _set_or_delete_marker_for_migrate_instances(cctxt, marker)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     instance.create()
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return fn(self, *args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 600, in create
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     db_inst = db.instance_create(self._context, updates)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 748, in instance_create
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return IMPL.instance_create(context, values)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 170, in wrapper
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(*args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     ectxt.value = e.inner_exc
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     self.force_reraise()
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     six.reraise(self.type_, self.value, self.tb)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(*args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1728, in instance_create
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     security_group_ensure_default(context)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4039, in security_group_ensure_default
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return _security_group_ensure_default(context)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4050, in _security_group_ensure_default
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     default_group = _security_group_get_by_names(context, ['default'])[0]
2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage TypeError: 'NoneType' object has no attribute '__getitem__'


mysql> select * from security_groups;
+---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
| created_at          | updated_at | deleted_at | id | name    | description | user_id                          | project_id                       | deleted |
+---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
| 2019-04-09 23:23:10 | NULL       | NULL       |  1 | default | default     | 7ed07a8869044a49b03a6e303e548198 | 1b57d6726e574f3db7fb94ceb3955591 |       0 |
| 2019-04-11 01:11:05 | NULL       | NULL       |  2 | default | default     | NULL                             | NULL                             |       0 |
| 2019-04-11 01:11:05 | NULL       | NULL       |  3 | default | default     | NULL                             | NULL                             |       0 |
+---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
3 rows in set (0.00 sec)

** Affects: nova
     Importance: High
         Status: Triaged

** Affects: nova/stein
     Importance: High
         Status: Triaged


** Tags: db nova-manage upgrade

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1824435

Title:
  fill_virtual_interface_list migration fails on second attempt

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) stein series:
  Triaged

Bug description:
  On attempting to run online_data_migrations on Stein for the second
  time (and beyond), fill_virtual_interface_list fails as below. I find
  two rows in the security_groups table which have name='default' and
  project_id NULL.

  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage [req-6cb533e2-58b5-41db-a455-29dae8efef31 - - - - -] Error attempting to run <function fill_virtual_interface_list at 0x7f1ed1b6f500>: TypeError: 'NoneType' object has no attribute '__getitem__'
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage Traceback (most recent call last):
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 686, in _run_migration
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     found, done = migration_meth(ctxt, count)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return fn(*args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     _set_or_delete_marker_for_migrate_instances(cctxt, marker)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     instance.create()
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return fn(self, *args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 600, in create
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     db_inst = db.instance_create(self._context, updates)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 748, in instance_create
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return IMPL.instance_create(context, values)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 170, in wrapper
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(*args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     ectxt.value = e.inner_exc
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     self.force_reraise()
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     six.reraise(self.type_, self.value, self.tb)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(*args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1728, in instance_create
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     security_group_ensure_default(context)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4039, in security_group_ensure_default
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return _security_group_ensure_default(context)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 227, in wrapped
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4050, in _security_group_ensure_default
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage     default_group = _security_group_get_by_names(context, ['default'])[0]
  2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage TypeError: 'NoneType' object has no attribute '__getitem__'

  
  mysql> select * from security_groups;
  +---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
  | created_at          | updated_at | deleted_at | id | name    | description | user_id                          | project_id                       | deleted |
  +---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
  | 2019-04-09 23:23:10 | NULL       | NULL       |  1 | default | default     | 7ed07a8869044a49b03a6e303e548198 | 1b57d6726e574f3db7fb94ceb3955591 |       0 |
  | 2019-04-11 01:11:05 | NULL       | NULL       |  2 | default | default     | NULL                             | NULL                             |       0 |
  | 2019-04-11 01:11:05 | NULL       | NULL       |  3 | default | default     | NULL                             | NULL                             |       0 |
  +---------------------+------------+------------+----+---------+-------------+----------------------------------+----------------------------------+---------+
  3 rows in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1824435/+subscriptions
Follow ups

[Bug 1824435] Re: fill_virtual_interface_list migration fails on second attempt
From: OpenStack Infra, 2019-10-26
[Bug 1824435] Re: fill_virtual_interface_list migration fails on second attempt
From: melanie witt, 2019-10-11
[Bug 1824435] Re: fill_virtual_interface_list migration fails on second attempt
From: Matt Riedemann, 2019-04-11