← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1385246] Re: Catch specific exceptions for _get_instance_nw_info

 

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1385246

Title:
  Catch specific exceptions for _get_instance_nw_info

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  Occasionally see the following logs:

  2014-10-19 17:29:54.170 27466 ERROR nova.compute.manager [-] [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] An error occurred while refreshing the network cache.
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] Traceback (most recent call last):
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/compute/manager.py", line 5327, in _heal_instance_info_cache
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     self._get_instance_nw_info(context, instance, use_slave=True)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/compute/manager.py", line 1233, in _get_instance_nw_info
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     instance)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/network/api.py", line 48, in wrapped
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return func(self, context, *args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/network/api.py", line 374, in get_instance_nw_info
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     result = self._get_instance_nw_info(context, instance)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/network/api.py", line 390, in _get_instance_nw_info
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/network/rpcapi.py", line 242, in get_instance_nw_info
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     host=host, project_id=project_id)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     retry=self.retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     timeout=timeout, retry=retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     retry=retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     raise result
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] InstanceNotFound_Remote: Instance dd50ac93-adf2-4e9d-915f-d8d1527d82e1 could not be found.
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] Traceback (most recent call last):
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/conductor/manager.py", line 400, in _object_dispatch
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return getattr(target, method)(context, *args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/objects/base.py", line 155, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     result = fn(cls, context, *args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/objects/virtual_interface.py", line 100, in get_by_instance_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/api.py", line 596, in virtual_interface_get_by_instance
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return f(*args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 155, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     instance_get_by_uuid(context, instance_uuid)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return f(*args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1719, in instance_get_by_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     columns_to_join=columns_to_join, use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1731, in _instance_get_by_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     raise exception.InstanceNotFound(instance_id=uuid)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] InstanceNotFound: Instance dd50ac93-adf2-4e9d-915f-d8d1527d82e1 could not be found.
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] Traceback (most recent call last):
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return func(*args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/network/manager.py", line 594, in get_instance_nw_info
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     context, instance_uuid, use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/objects/base.py", line 153, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     args, kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 346, in object_class_action
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     objver=objver, args=args, kwargs=kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     retry=self.retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     timeout=timeout, retry=retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     retry=retry)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     raise result
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] InstanceNotFound_Remote: Instance dd50ac93-adf2-4e9d-915f-d8d1527d82e1 could not be found.
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] Traceback (most recent call last):
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/conductor/manager.py", line 400, in _object_dispatch
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return getattr(target, method)(context, *args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/objects/base.py", line 155, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     result = fn(cls, context, *args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/objects/virtual_interface.py", line 100, in get_by_instance_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/api.py", line 596, in virtual_interface_get_by_instance
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return f(*args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 155, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     instance_get_by_uuid(context, instance_uuid)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     return f(*args, **kwargs)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1719, in instance_get_by_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     columns_to_join=columns_to_join, use_slave=use_slave)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1731, in _instance_get_by_uuid
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1]     raise exception.InstanceNotFound(instance_id=uuid)
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] InstanceNotFound: Instance dd50ac93-adf2-4e9d-915f-d8d1527d82e1 could not be found.
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 
  2014-10-19 17:29:54.170 27466 TRACE nova.compute.manager [instance: dd50ac93-adf2-4e9d-915f-d8d1527d82e1] 

  We can catch a specifc exception here and not have the ugly log
  traces.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1385246/+subscriptions


References