yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1445698] [NEW] _update_usage_from_migrations does not handling InstanceNotFound and cause compute service restart to fail

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Christine Wang <ijuwang@xxxxxxxxxx>
Date: Sat, 18 Apr 2015 01:19:01 -0000
Reply-to: Bug 1445698 <1445698@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Due to bug #1445674, Migration object was not set to error if there is a
resize failure.

Later on, if we delete the instance, the Migration object will continue
to exist.

If we were to restart compute service, it will fail to start since
_update_usage_from_migrations

Version:
commit 095e9398ecf69ffdaeb929287d5f5f9a38257361
Merge: 6029860 0e28a5f
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Fri Apr 17 19:47:18 2015 +0000

    Merge "fixed tests in test_iptables_network to work with random
PYTHONHASHSE

commit 6029860ffa0f2500505d1894f5bbb9ca717a8232
Merge: 760fba5 5bfe303
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Fri Apr 17 19:46:50 2015 +0000

    Merge "refactored tests in test_objects to pass with random
PYTHONHASHSEED"

commit 760fba535b2eb17243a39af9fea70e8dbcdbe713
Merge: 1248353 78883fa
[root@ip9-114-195-109 nova]# git log -1
commit 20cb0745550fc6bbd9e789caa7fdbf9669b2d24d
Merge: 095e939 56f355e
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Fri Apr 17 1


2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 445, in _update_available_resource
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     self._update_usage_from_migrations(context, resources, migrations)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 708, in _update_usage_from_migrations
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     instance = migration.instance
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/migration.py", line 80, in instance
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     return objects.Instance.get_by_uuid(self._context, self.instance_uuid)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 564, in get_by_uuid
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     use_slave=use_slave)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 651, in instance_get_by_uuid
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     columns_to_join, use_slave=use_slave)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1744, in instance_get_by_uuid
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     columns_to_join=columns_to_join, use_slave=use_slave)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1756, in _instance_get_by_uuid
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     raise exception.InstanceNotFound(instance_id=uuid)
2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup InstanceNotFound: Instance b8ddd534-f114-4ea6-9833-eeb64a8bfc49 could not be found.


    def _update_usage_from_migrations(self, context, resources, migrations):

        self.tracked_migrations.clear()

        filtered = {}

        # do some defensive filtering against bad migrations records in the
        # database:
        for migration in migrations:
            instance = migration.instance                               <--- getting InstanceNotFound

            if not instance:
                # migration referencing deleted instance
                continue

            uuid = instance.uuid

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1445698

Title:
  _update_usage_from_migrations does not handling InstanceNotFound and
  cause compute service restart to fail

Status in OpenStack Compute (Nova):
  New

Bug description:
  Due to bug #1445674, Migration object was not set to error if there is
  a resize failure.

  Later on, if we delete the instance, the Migration object will
  continue to exist.

  If we were to restart compute service, it will fail to start since
  _update_usage_from_migrations

  Version:
  commit 095e9398ecf69ffdaeb929287d5f5f9a38257361
  Merge: 6029860 0e28a5f
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Fri Apr 17 19:47:18 2015 +0000

      Merge "fixed tests in test_iptables_network to work with random
  PYTHONHASHSE

  commit 6029860ffa0f2500505d1894f5bbb9ca717a8232
  Merge: 760fba5 5bfe303
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Fri Apr 17 19:46:50 2015 +0000

      Merge "refactored tests in test_objects to pass with random
  PYTHONHASHSEED"

  commit 760fba535b2eb17243a39af9fea70e8dbcdbe713
  Merge: 1248353 78883fa
  [root@ip9-114-195-109 nova]# git log -1
  commit 20cb0745550fc6bbd9e789caa7fdbf9669b2d24d
  Merge: 095e939 56f355e
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Fri Apr 17 1

  
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 445, in _update_available_resource
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     self._update_usage_from_migrations(context, resources, migrations)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 708, in _update_usage_from_migrations
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     instance = migration.instance
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/migration.py", line 80, in instance
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     return objects.Instance.get_by_uuid(self._context, self.instance_uuid)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 564, in get_by_uuid
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     use_slave=use_slave)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 651, in instance_get_by_uuid
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     columns_to_join, use_slave=use_slave)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1744, in instance_get_by_uuid
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     columns_to_join=columns_to_join, use_slave=use_slave)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1756, in _instance_get_by_uuid
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup     raise exception.InstanceNotFound(instance_id=uuid)
  2015-04-17 05:56:58.402 22873 TRACE nova.openstack.common.threadgroup InstanceNotFound: Instance b8ddd534-f114-4ea6-9833-eeb64a8bfc49 could not be found.

  
      def _update_usage_from_migrations(self, context, resources, migrations):

          self.tracked_migrations.clear()

          filtered = {}

          # do some defensive filtering against bad migrations records in the
          # database:
          for migration in migrations:
              instance = migration.instance                               <--- getting InstanceNotFound

              if not instance:
                  # migration referencing deleted instance
                  continue

              uuid = instance.uuid

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1445698/+subscriptions
Follow ups

[Bug 1445698] [NEW] _update_usage_from_migrations does not handling InstanceNotFound and cause compute service restart to fail
From: Christine Wang, 2015-04-18
References

[Bug 1445698] [NEW] _update_usage_from_migrations does not handling InstanceNotFound and cause compute service restart to fail
From: Christine Wang, 2015-04-18