← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1444439] Re: Resource tracker: unable to start nova compute

 

** Also affects: nova/kilo
   Importance: Undecided
       Status: New

** Changed in: nova/kilo
    Milestone: None => kilo-rc2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1444439

Title:
  Resource tracker: unable to start nova compute

Status in OpenStack Compute (Nova):
  Fix Committed
Status in OpenStack Compute (nova) kilo series:
  New

Bug description:
  After a failure of the resize and a deletion of the instance. I am
  unable to restart the nova compute due to the exception below. The
  instance was deleted via nova api.

  The DB is as follows:
  mysql> select * from migrations;
  +---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
  | created_at          | updated_at          | deleted_at | id | source_compute   | dest_compute     | dest_host     | status         | instance_uuid                        | old_instance_type_id | new_instance_type_id | source_node      | dest_node        | deleted |
  +---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
  | 2015-04-15 09:44:02 | 2015-04-15 09:44:08 | NULL       |  1 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | post-migrating | 42264e24-1385-41f1-8dfc-120a1891ab05 |                   10 |                   11 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 09:48:13 | 2015-04-15 10:19:48 | NULL       |  2 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | reverted       | fcab4bde-d93e-4d79-ae35-9d1306da10a4 |                   10 |                   11 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:23:56 | 2015-04-15 10:24:03 | NULL       |  3 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | post-migrating | d074bbc0-b912-4c85-a02b-aabf56d45f0b |                   10 |                   11 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:27:45 | 2015-04-15 10:28:16 | NULL       |  4 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | reverted       | 21e59c96-fa2f-45e3-9070-e982a2dafea6 |                   10 |                   11 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:28:43 | 2015-04-15 10:29:16 | NULL       |  5 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirming     | 21e59c96-fa2f-45e3-9070-e982a2dafea6 |                   10 |                   11 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:35:15 | 2015-04-15 10:53:16 | NULL       |  6 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirmed      | 4abd75b5-bb91-4ce7-a928-2a96941ea9cb |                   10 |                   14 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:35:39 | 2015-04-15 10:53:17 | NULL       |  7 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirmed      | 5e01bddb-3978-4f6f-a4d3-6d24ed31afa4 |                   14 |                   10 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  | 2015-04-15 10:55:01 | 2015-04-15 10:55:02 | NULL       |  8 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | migrating      | 20017567-5c83-4918-b269-525169009026 |                   10 |                   15 | domain-c167(DVS) | domain-c167(DVS) |       0 |
  +---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
  8 rows in set (0.00 sec)

  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 145, in wait
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/service.py", line 497, in run_service
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     service.start()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/service.py", line 183, in start
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 1287, in pre_start_hook
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 6236, in update_available_resource
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 402, in update_available_resource
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 445, in _update_available_resource
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     self._update_usage_from_migrations(context, resources, migrations)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/resource_tracker.py", line 709, in _update_usage_from_migrations
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     instance = migration.instance
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/objects/migration.py", line 80, in instance
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return objects.Instance.get_by_uuid(self._context, self.instance_uuid)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     args, kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 325, in object_class_action
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     objver=objver, args=args, kwargs=kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     retry=self.retry)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     timeout=timeout, retry=retry)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     retry=retry)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     raise result
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup InstanceNotFound_Remote: Instance 42264e24-1385-41f1-8dfc-120a1891ab05 could not be found.
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/conductor/manager.py", line 423, in _object_dispatch
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return getattr(target, method)(*args, **kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/objects/base.py", line 163, in wrapper
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/objects/instance.py", line 564, in get_by_uuid
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     use_slave=use_slave)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/db/api.py", line 651, in instance_get_by_uuid
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     columns_to_join, use_slave=use_slave)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1744, in instance_get_by_uuid
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     columns_to_join=columns_to_join, use_slave=use_slave)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1756, in _instance_get_by_uuid
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup     raise exception.InstanceNotFound(instance_id=uuid)
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup InstanceNotFound: Instance 42264e24-1385-41f1-8dfc-120a1891ab05 could not be found.
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup 
  2015-04-15 04:47:04.824 INFO oslo_vmware.api [req-2d7c5d01-438c-494c-a382-b7c55b71d8be None None] Logging out and terminating the current session with ID = f4fcb.
  nicira@Ubuntu1404Server:~/devstack$

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1444439/+subscriptions


References