yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1412285] Re: InstanceInfoCacheNotFound exception

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: John Garbutt <john@xxxxxxxxxxxxxxx>
Date: Thu, 07 Jul 2016 13:02:51 -0000
Reply-to: Bug 1412285 <1412285@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Looks like this got fixed, marking it as invalid.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1412285

Title:
  InstanceInfoCacheNotFound exception

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Operating steps:
  1. nova boot a instance named 'A' on host named 'hostxx'.
  2. use 'systemctl stop openstack-nova-compute' command to stop nova-compute service on hostxx.
  3. use 'nova delete' command delete instance A, and the result is successful.
  4. Then use 'systemctl start openstack-nova-compute' command to start nova-compute service on hostxx.
  5. wait 30 minutes, the periodic task named '_cleanup_running_deleted_instances' will execute to delete the instance A on hostxx and the exception InstanceInfoCacheNotFound occurs. (You can modify the value of 'running_deleted_instance_poll_interval' in file /etc/nova/nova.conf to reduce the waiting time)

  the nova-compute.log on hostxx:
  root@lxlcompute1 ~]# tail -100 /var/log/nova/nova-compute.log
  2015-03-17 15:55:36.764 17764 WARNING nova.compute.manager [-] Found 0 in the database and 1 on the hypervisor.
  2015-03-17 15:56:00.963 17764 INFO nova.compute.manager [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Destroying instance with name label 'instance-00000056' which is marked as DELETED but still present on host.
  2015-03-17 15:56:00.964 17764 AUDIT nova.compute.manager [req-2d912481-6e91-4713-a62a-694232c4e58c None None] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Terminating instance
  2015-03-17 15:56:01.263 17764 INFO nova.compute.manager [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] VM Stopped (Lifecycle Event)
  2015-03-17 15:56:01.272 17764 INFO nova.virt.libvirt.driver [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Instance destroyed successfully.
  2015-03-17 15:56:01.854 17764 ERROR nova.network.api [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Failed storing info cache
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Traceback (most recent call last):
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 81, in update_instance_cache_with_nw_info
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     ic.save(update_cells=update_cells)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 142, in wrapper
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     ctxt, self, fn.__name__, args, kwargs)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 430, in object_action
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     objmethod=objmethod, args=args, kwargs=kwargs)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     wait_for_reply=True, timeout=timeout)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     timeout=timeout)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     raise result
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] InstanceInfoCacheNotFound_Remote: Info cache for instance b4217413-f4ef-4e0c-84a6-3e4739bb6158 could not be found.
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Traceback (most recent call last):
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 598, in _object_dispatch
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     return getattr(target, method)(context, *args, **kwargs)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 151, in wrapper
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     return fn(self, ctxt, *args, **kwargs)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line 91, in save
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     {'network_info': nw_info_json})
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 879, in instance_info_cache_update
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     return IMPL.instance_info_cache_update(context, instance_uuid, values)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     return f(*args, **kwargs)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2353, in instance_info_cache_update
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]     instance_uuid=instance_uuid)
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] InstanceInfoCacheNotFound: Info cache for instance b4217413-f4ef-4e0c-84a6-3e4739bb6158 could not be found.
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.854 17764 TRACE nova.network.api [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158]
  2015-03-17 15:56:01.855 17764 ERROR nova.compute.manager [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Failed to deallocate network for instance.
  2015-03-17 15:56:01.982 17764 ERROR root [-] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2322, in _shutdown_instance\n    block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 971, in destroy\n    destroy_disks)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1050, in cleanup\n    volume_meta[\'properties\'] = self._get_volume_metadata(context, connection_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3088, in _get_volume_metadata\n    raise exception.InvalidBDMVolume(id=volume_id)\n', 'InvalidBDMVolume: Block Device Mapping is Invalid: failed to get volume 7fad9bb5-8b65-4647-af85-91a72252862f.\n']
  2015-03-17 15:56:01.982 17764 WARNING nova.compute.manager [-] [instance: b4217413-f4ef-4e0c-84a6-3e4739bb6158] Periodic cleanup failed to delete instance: Info cache for instance b4217413-f4ef-4e0c-84a6-3e4739bb6158 could not be found.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 598, in _object_dispatch
      return getattr(target, method)(context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 151, in wrapper
      return fn(self, ctxt, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line 91, in save
      {'network_info': nw_info_json})

    File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 879, in instance_info_cache_update
      return IMPL.instance_info_cache_update(context, instance_uuid, values)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2353, in instance_info_cache_update
      instance_uuid=instance_uuid)

  InstanceInfoCacheNotFound: Info cache for instance b4217413-f4ef-4e0c-
  84a6-3e4739bb6158 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1412285/+subscriptions
References

[Bug 1412285] [NEW] InstanceInfoCacheNotFound exception
From: Rong Han ZTE, 2015-01-19