← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #65688

[Bug 1702959] [NEW] MigrationNotFound in multi-cell setup doing server external events processing

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I noticed this in the devstack change testing multi-cell, in the neutron
multinode job:

http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-
multinode-full-ubuntu-xenial-
nv/ee3e9b6/logs/screen-n-api.txt.gz?level=TRACE#_Jul_05_20_55_53_749696

We're getting migration not found because we're not targeted to a cell
to find the migration:

Jul 05 20:55:53.749696 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions [None req-d4ca5b2e-140c-4331-88eb-50bae2de2230 service nova] Unexpected exception in API method: MigrationNotFound: Migration 15 could not be found.
Jul 05 20:55:53.749806 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions Traceback (most recent call last):
Jul 05 20:55:53.749888 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 336, in wrapped
Jul 05 20:55:53.749965 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
Jul 05 20:55:53.750053 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 108, in wrapper
Jul 05 20:55:53.750134 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
Jul 05 20:55:53.750222 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/server_external_events.py", line 120, in create
Jul 05 20:55:53.750303 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     context, accepted_instances, mappings, accepted_events)
Jul 05 20:55:53.750438 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4046, in external_instance_event
Jul 05 20:55:53.750564 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     for host in self._get_relevant_hosts(context, instance):
Jul 05 20:55:53.750647 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4072, in _get_relevant_hosts
Jul 05 20:55:53.750722 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     migration = objects.Migration.get_by_id(context, migration_id)
Jul 05 20:55:53.750798 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jul 05 20:55:53.750873 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
Jul 05 20:55:53.750949 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/migration.py", line 106, in get_by_id
Jul 05 20:55:53.751023 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     db_migration = db.migration_get(context, migration_id)
Jul 05 20:55:53.751107 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 535, in migration_get
Jul 05 20:55:53.751187 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return IMPL.migration_get(context, migration_id)
Jul 05 20:55:53.751264 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped
Jul 05 20:55:53.751338 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
Jul 05 20:55:53.751414 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4719, in migration_get
Jul 05 20:55:53.751492 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     raise exception.MigrationNotFound(migration_id=id)
Jul 05 20:55:53.751568 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions MigrationNotFound: Migration 15 could not be found.
Jul 05 20:55:53.751640 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions 

https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L4301

We need to use the mappings dict passed to external_instance_event to
mutate the context while looking up the migration.

** Affects: nova
     Importance: High
         Status: Triaged


** Tags: api cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1702959

Title:
  MigrationNotFound in multi-cell setup doing server external events
  processing

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  I noticed this in the devstack change testing multi-cell, in the
  neutron multinode job:

  http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-
  multinode-full-ubuntu-xenial-
  nv/ee3e9b6/logs/screen-n-api.txt.gz?level=TRACE#_Jul_05_20_55_53_749696

  We're getting migration not found because we're not targeted to a cell
  to find the migration:

  Jul 05 20:55:53.749696 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions [None req-d4ca5b2e-140c-4331-88eb-50bae2de2230 service nova] Unexpected exception in API method: MigrationNotFound: Migration 15 could not be found.
  Jul 05 20:55:53.749806 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions Traceback (most recent call last):
  Jul 05 20:55:53.749888 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 336, in wrapped
  Jul 05 20:55:53.749965 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  Jul 05 20:55:53.750053 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 108, in wrapper
  Jul 05 20:55:53.750134 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  Jul 05 20:55:53.750222 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/server_external_events.py", line 120, in create
  Jul 05 20:55:53.750303 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     context, accepted_instances, mappings, accepted_events)
  Jul 05 20:55:53.750438 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4046, in external_instance_event
  Jul 05 20:55:53.750564 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     for host in self._get_relevant_hosts(context, instance):
  Jul 05 20:55:53.750647 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4072, in _get_relevant_hosts
  Jul 05 20:55:53.750722 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     migration = objects.Migration.get_by_id(context, migration_id)
  Jul 05 20:55:53.750798 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Jul 05 20:55:53.750873 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  Jul 05 20:55:53.750949 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/migration.py", line 106, in get_by_id
  Jul 05 20:55:53.751023 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     db_migration = db.migration_get(context, migration_id)
  Jul 05 20:55:53.751107 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 535, in migration_get
  Jul 05 20:55:53.751187 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return IMPL.migration_get(context, migration_id)
  Jul 05 20:55:53.751264 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped
  Jul 05 20:55:53.751338 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  Jul 05 20:55:53.751414 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4719, in migration_get
  Jul 05 20:55:53.751492 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     raise exception.MigrationNotFound(migration_id=id)
  Jul 05 20:55:53.751568 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions MigrationNotFound: Migration 15 could not be found.
  Jul 05 20:55:53.751640 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions 

  https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L4301

  We need to use the mappings dict passed to external_instance_event to
  mutate the context while looking up the migration.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1702959/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next