yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1618822] Re: downgrade the exception log in update_instance_cache_with_nw_info

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 08 Aug 2017 15:21:28 -0000
Reply-to: Bug 1618822 <1618822@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** This bug is no longer a duplicate of bug 1632403
   InstanceInfoCacheNotFound: Info cache for instance 7e3be027-0303-4ed8-8dfc-5c19a419901c could not be found

** Changed in: nova
   Importance: Undecided => Medium

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Changed in: nova/ocata
       Status: New => Confirmed

** Changed in: nova/ocata
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1618822

Title:
  downgrade the exception log in update_instance_cache_with_nw_info

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) ocata series:
  Confirmed

Bug description:
  when compute node is down, nova api can accept delete action and delete the instance
  then during compute start up, periodic task will check whether the instance is deleted in api layer
   then performance actions

  saw those logs because actually the info case is already deleted so
  it's a valid case and should not report this exception ,just a log
  info should be fine

  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 50, in update_instance_cache_with_nw_info
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     ic.save(update_cells=update_cells)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in wrapper
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     ctxt, self, fn.__name__, args, kwargs)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in object_action
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     objmethod=objmethod, args=args, kwargs=kwargs)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     retry=self.retry)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     timeout=timeout, retry=retry)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
  ages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     retry=retry)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     raise result
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound_Remote: Info cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447, in _object_dispatch
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     return getattr(target, method)(*args, **kwargs)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     return fn(self, *args, **kwargs)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line 98, in save
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     {'network_info': nw_info_json})
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in instance_info_cache_update
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     return IMPL.instance_info_cache_update(context, instance_uuid, values)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     return f(*args, **kwargs)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2684, in instance_info_cache_update
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]     instance_uuid=instance_uuid)
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound: Info cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
  2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71]


  
  2016-08-31 10:13:57.101 25191 WARNING nova.compute.manager [req-7801f16c-8845-4825-a18e-9806d142882c - - - - -] [instance: d6a78566-0f7d-4173-b35a-b45d2054ba71] Periodic cleanup failed to delete instance: Info cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in wrapper
      return fn(self, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line 98, in save
      {'network_info': nw_info_json})

    File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in instance_info_cache_update
      return IMPL.instance_info_cache_update(context, instance_uuid, values)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper
      return f(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2684, in instance_info_cache_update
      instance_uuid=instance_uuid)

  InstanceInfoCacheNotFound: Info cache for instance d6a78566-0f7d-4173
  -b35a-b45d2054ba71 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1618822/+subscriptions
References

[Bug 1618822] [NEW] downgrade the exception log in update_instance_cache_with_nw_info
From: jichenjc, 2016-08-31