yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1702959] Re: MigrationNotFound in multi-cell setup doing server external events processing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1702959@xxxxxxxxxxxxxxxxxx>
Date: Fri, 21 Jul 2017 00:13:40 -0000
Reply-to: Bug 1702959 <1702959@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/445142
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8be61eb39dc9f605ad853e5697a8f4bf73b025b
Submitter: Jenkins
Branch:    master

commit b8be61eb39dc9f605ad853e5697a8f4bf73b025b
Author: Matthew Booth <mbooth@xxxxxxxxxx>
Date:   Mon Mar 13 18:28:17 2017 +0000

    Fix and optimize external_events for multiple cells
    
    server_external_events was previously making an api db query and a
    cell db query for every instance reference. We improve this by
    making exactly 1 api db query to fetch all instance mappings, and then
    1 cell db query per cell to fetch all relevant instances from that
    cell. Further, it wasn't properly handling the case where events
    were delivered in one request for multiple instances across cells,
    which this also fixes.
    
    We also document an obtuse edge condition in
    ComputeAPI.external_instance_event which will cause the current code
    to break when we support migration between cells.
    
    Note this includes a tweak to the SingleCellSimple fixture to mock out
    the new InstanceMappingList method we use, as well as a fix to the other
    InstanceMapping mock, which was returning mappings with bogus instance
    uuids. This patch relies on the results of those being realistic and
    thus requires those changes.
    
    Closes-Bug: #1702959
    
    Co-Authored-By: Dan Smith <dms@xxxxxxxxxxxxx>
    
    Change-Id: If59453f1899e99040c554bcb9ad54c8a506adc56


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1702959

Title:
  MigrationNotFound in multi-cell setup doing server external events
  processing

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  I noticed this in the devstack change testing multi-cell, in the
  neutron multinode job:

  http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-
  multinode-full-ubuntu-xenial-
  nv/ee3e9b6/logs/screen-n-api.txt.gz?level=TRACE#_Jul_05_20_55_53_749696

  We're getting migration not found because we're not targeted to a cell
  to find the migration:

  Jul 05 20:55:53.749696 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions [None req-d4ca5b2e-140c-4331-88eb-50bae2de2230 service nova] Unexpected exception in API method: MigrationNotFound: Migration 15 could not be found.
  Jul 05 20:55:53.749806 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions Traceback (most recent call last):
  Jul 05 20:55:53.749888 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 336, in wrapped
  Jul 05 20:55:53.749965 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  Jul 05 20:55:53.750053 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 108, in wrapper
  Jul 05 20:55:53.750134 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  Jul 05 20:55:53.750222 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/server_external_events.py", line 120, in create
  Jul 05 20:55:53.750303 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     context, accepted_instances, mappings, accepted_events)
  Jul 05 20:55:53.750438 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4046, in external_instance_event
  Jul 05 20:55:53.750564 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     for host in self._get_relevant_hosts(context, instance):
  Jul 05 20:55:53.750647 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 4072, in _get_relevant_hosts
  Jul 05 20:55:53.750722 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     migration = objects.Migration.get_by_id(context, migration_id)
  Jul 05 20:55:53.750798 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Jul 05 20:55:53.750873 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  Jul 05 20:55:53.750949 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/migration.py", line 106, in get_by_id
  Jul 05 20:55:53.751023 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     db_migration = db.migration_get(context, migration_id)
  Jul 05 20:55:53.751107 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/api.py", line 535, in migration_get
  Jul 05 20:55:53.751187 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return IMPL.migration_get(context, migration_id)
  Jul 05 20:55:53.751264 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped
  Jul 05 20:55:53.751338 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  Jul 05 20:55:53.751414 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4719, in migration_get
  Jul 05 20:55:53.751492 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions     raise exception.MigrationNotFound(migration_id=id)
  Jul 05 20:55:53.751568 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions MigrationNotFound: Migration 15 could not be found.
  Jul 05 20:55:53.751640 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 devstack@n-api.service[27811]: ERROR nova.api.openstack.extensions 

  https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L4301

  We need to use the mappings dict passed to external_instance_event to
  mutate the context while looking up the migration.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1702959/+subscriptions
References

[Bug 1702959] [NEW] MigrationNotFound in multi-cell setup doing server external events processing
From: Matt Riedemann, 2017-07-07