yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1980582] [NEW] Upgrade from stein to train fails db migration fill_virtual_interface_list

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Patrick Oberdorf <1980582@xxxxxxxxxxxxxxxxxx>
Date: Sun, 03 Jul 2022 00:25:26 -0000
Reply-to: Bug 1980582 <1980582@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We are upgrading an old Openstack stein to train but we are facing a
strange error:

Running batches of 50 until complete
Error attempting to run <function fill_virtual_interface_list at 0x7f7593af2f28>
+---------------------------------------------+--------------+-----------+
|                  Migration                  | Total Needed | Completed |
+---------------------------------------------+--------------+-----------+
|         create_incomplete_consumers         |      0       |     0     |
| delete_build_requests_with_no_instance_uuid |      0       |     0     |
|         fill_virtual_interface_list         |      0       |     0     |
|             migrate_empty_ratio             |      0       |     0     |
|          migrate_keypairs_to_api_db         |      0       |     0     |
|       migrate_quota_classes_to_api_db       |      0       |     0     |
|        migrate_quota_limits_to_api_db       |      0       |     0     |
|          migration_migrate_to_uuid          |      0       |     0     |
|     populate_missing_availability_zones     |      0       |     0     |
|          populate_queued_for_delete         |      0       |     0     |
|               populate_user_id              |      0       |     0     |
|                populate_uuids               |      0       |     0     |
|     service_uuids_online_data_migration     |      0       |     0     |
+---------------------------------------------+--------------+-----------+
Some migrations failed unexpectedly. Check log for details.


2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage [req-5c6fda26-231f-4040-871f-cfd80296c94b - - - - -] Error attempting to run <function fill_virtual_interface_list at 0x7f7593af2f28>: TypeError: 'NoneType' object is not subscriptable
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage Traceback (most recent call last):
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 686, in _run_migration
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     found, done = migration_meth(ctxt, count)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return fn(*args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     _set_or_delete_marker_for_migrate_instances(cctxt, marker)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     instance.create()
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return fn(self, *args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/instance.py", line 600, in create
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     db_inst = db.instance_create(self._context, updates)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/api.py", line 748, in instance_create
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return IMPL.instance_create(context, values)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 192, in wrapper
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(*args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     ectxt.value = e.inner_exc
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     self.force_reraise()
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     six.reraise(self.type_, self.value, self.tb)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     raise value
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(*args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 1793, in instance_create
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     default_group = security_group_ensure_default(context)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 4107, in security_group_ensure_default
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return _security_group_ensure_default(context)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 4118, in _security_group_ensure_default
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     default_group = _security_group_get_by_names(context, ['default'])[0]
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage TypeError: 'NoneType' object is not subscriptable
2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage


So without fixing this we are stuck on stein release. Do you have a hint what I can check? Thanks in advanced.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1980582

Title:
  Upgrade from stein to train fails db migration
  fill_virtual_interface_list

Status in OpenStack Compute (nova):
  New

Bug description:
  We are upgrading an old Openstack stein to train but we are facing a
  strange error:

  Running batches of 50 until complete
  Error attempting to run <function fill_virtual_interface_list at 0x7f7593af2f28>
  +---------------------------------------------+--------------+-----------+
  |                  Migration                  | Total Needed | Completed |
  +---------------------------------------------+--------------+-----------+
  |         create_incomplete_consumers         |      0       |     0     |
  | delete_build_requests_with_no_instance_uuid |      0       |     0     |
  |         fill_virtual_interface_list         |      0       |     0     |
  |             migrate_empty_ratio             |      0       |     0     |
  |          migrate_keypairs_to_api_db         |      0       |     0     |
  |       migrate_quota_classes_to_api_db       |      0       |     0     |
  |        migrate_quota_limits_to_api_db       |      0       |     0     |
  |          migration_migrate_to_uuid          |      0       |     0     |
  |     populate_missing_availability_zones     |      0       |     0     |
  |          populate_queued_for_delete         |      0       |     0     |
  |               populate_user_id              |      0       |     0     |
  |                populate_uuids               |      0       |     0     |
  |     service_uuids_online_data_migration     |      0       |     0     |
  +---------------------------------------------+--------------+-----------+
  Some migrations failed unexpectedly. Check log for details.

  
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage [req-5c6fda26-231f-4040-871f-cfd80296c94b - - - - -] Error attempting to run <function fill_virtual_interface_list at 0x7f7593af2f28>: TypeError: 'NoneType' object is not subscriptable
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage Traceback (most recent call last):
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 686, in _run_migration
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     found, done = migration_meth(ctxt, count)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return fn(*args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     _set_or_delete_marker_for_migrate_instances(cctxt, marker)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     instance.create()
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return fn(self, *args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/objects/instance.py", line 600, in create
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     db_inst = db.instance_create(self._context, updates)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/api.py", line 748, in instance_create
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return IMPL.instance_create(context, values)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 192, in wrapper
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(*args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     ectxt.value = e.inner_exc
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     self.force_reraise()
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     six.reraise(self.type_, self.value, self.tb)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     raise value
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(*args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 1793, in instance_create
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     default_group = security_group_ensure_default(context)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 4107, in security_group_ensure_default
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return _security_group_ensure_default(context)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     return f(context, *args, **kwargs)
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage   File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 4118, in _security_group_ensure_default
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage     default_group = _security_group_get_by_names(context, ['default'])[0]
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage TypeError: 'NoneType' object is not subscriptable
  2022-07-03 00:23:13.241 26829 ERROR nova.cmd.manage

  
  So without fixing this we are stuck on stein release. Do you have a hint what I can check? Thanks in advanced.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1980582/+subscriptions
Follow ups

[Bug 1980582] Re: Upgrade from stein to train fails db migration fill_virtual_interface_list
From: Sylvain Bauza, 2022-07-05