yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1473909] [NEW] Error message during nova delete (Esxi based devstack setup using ovsvapp sollution)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mh Raies <1473909@xxxxxxxxxxxxxxxxxx>
Date: Mon, 13 Jul 2015 09:38:30 -0000
Reply-to: Bug 1473909 <1473909@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am trying "https://github.com/openstack/networking-vsphere/tree/master/devstack"; for OVSvApp solution , consisting 3 DVS.
1. Trunk DVS
2. Management DVS
3. Uplink DVS

I am using Esxi based devstack setup using vCenter. Also I am working
with stable kilo.

I could successfully boot an instance using nova boot.

When I delete same instance using nova delete, API request is successful.
VM deletes after a long. But in the mean time following error occurs - 



2015-07-13 21:53:44.193 ERROR nova.network.base_api [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Failed storing info cache
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Traceback (most recent call last):
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/network/base_api.py", line 49, in update_instance_cache_with_nw_info
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     ic.save(update_cells=update_cells)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/base.py", line 192, in wrapper
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     self._context, self, fn.__name__, args, kwargs)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 340, in object_action
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     objmethod=objmethod, args=args, kwargs=kwargs)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     retry=self.retry)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     timeout=timeout, retry=retry)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     retry=retry)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     raise result
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] InstanceInfoCacheNotFound_Remote: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Traceback (most recent call last):
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return getattr(target, method)(*args, **kwargs)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/base.py", line 208, in wrapper
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return fn(self, *args, **kwargs)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 95, in save
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     {'network_info': nw_info_json})
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/api.py", line 888, in instance_info_cache_update
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return IMPL.instance_info_cache_update(context, instance_uuid, values)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return f(*args, **kwargs)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2624, in instance_info_cache_update
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     instance_uuid=instance_uuid)
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] InstanceInfoCacheNotFound: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
2015-07-13 21:53:44.195 ERROR nova.compute.manager [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Failed to deallocate network for instance.
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2015-07-13 21:53:44.516 WARNING nova.compute.manager [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Periodic cleanup failed to delete instance: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
Traceback (most recent call last):

  File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/opt/stack/nova/nova/objects/base.py", line 208, in wrapper
    return fn(self, *args, **kwargs)

  File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 95, in save
    {'network_info': nw_info_json})

  File "/opt/stack/nova/nova/db/api.py", line 888, in instance_info_cache_update
    return IMPL.instance_info_cache_update(context, instance_uuid, values)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
    return f(*args, **kwargs)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2624, in instance_info_cache_update
    instance_uuid=instance_uuid)

InstanceInfoCacheNotFound: Info cache for instance 06a3de55-285d-4d0d-
953e-7f99aed28e95 could not be found.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1473909

Title:
  Error message during nova delete (Esxi based devstack setup using
  ovsvapp sollution)

Status in OpenStack Compute (nova):
  New

Bug description:
  I am trying "https://github.com/openstack/networking-vsphere/tree/master/devstack"; for OVSvApp solution , consisting 3 DVS.
  1. Trunk DVS
  2. Management DVS
  3. Uplink DVS

  I am using Esxi based devstack setup using vCenter. Also I am working
  with stable kilo.

  I could successfully boot an instance using nova boot.

  When I delete same instance using nova delete, API request is successful.
  VM deletes after a long. But in the mean time following error occurs - 



  
  2015-07-13 21:53:44.193 ERROR nova.network.base_api [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Failed storing info cache
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Traceback (most recent call last):
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/network/base_api.py", line 49, in update_instance_cache_with_nw_info
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     ic.save(update_cells=update_cells)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/base.py", line 192, in wrapper
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     self._context, self, fn.__name__, args, kwargs)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 340, in object_action
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     objmethod=objmethod, args=args, kwargs=kwargs)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     retry=self.retry)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     timeout=timeout, retry=retry)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     retry=retry)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     raise result
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] InstanceInfoCacheNotFound_Remote: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Traceback (most recent call last):
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return getattr(target, method)(*args, **kwargs)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/base.py", line 208, in wrapper
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return fn(self, *args, **kwargs)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 95, in save
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     {'network_info': nw_info_json})
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/api.py", line 888, in instance_info_cache_update
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return IMPL.instance_info_cache_update(context, instance_uuid, values)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     return f(*args, **kwargs)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2624, in instance_info_cache_update
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]     instance_uuid=instance_uuid)
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] InstanceInfoCacheNotFound: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.193 TRACE nova.network.base_api [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95]
  2015-07-13 21:53:44.195 ERROR nova.compute.manager [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Failed to deallocate network for instance.
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
    InsecureRequestWarning)
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
    InsecureRequestWarning)
  /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
    InsecureRequestWarning)
  2015-07-13 21:53:44.516 WARNING nova.compute.manager [req-760e73b5-9815-441d-931e-c0a57f8d32f3 None None] [instance: 06a3de55-285d-4d0d-953e-7f99aed28e95] Periodic cleanup failed to delete instance: Info cache for instance 06a3de55-285d-4d0d-953e-7f99aed28e95 could not be found.
  Traceback (most recent call last):

    File "/opt/stack/nova/nova/conductor/manager.py", line 422, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/opt/stack/nova/nova/objects/base.py", line 208, in wrapper
      return fn(self, *args, **kwargs)

    File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 95, in save
      {'network_info': nw_info_json})

    File "/opt/stack/nova/nova/db/api.py", line 888, in instance_info_cache_update
      return IMPL.instance_info_cache_update(context, instance_uuid, values)

    File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
      return f(*args, **kwargs)

    File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2624, in instance_info_cache_update
      instance_uuid=instance_uuid)

  InstanceInfoCacheNotFound: Info cache for instance 06a3de55-285d-4d0d-
  953e-7f99aed28e95 could not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1473909/+subscriptions
Follow ups

[Bug 1473909] Re: Error message during nova delete (Esxi based devstack setup using ovsvapp sollution)
From: Romil Gupta, 2015-11-04