yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1587356] [NEW] nova raises "InstanceNotFound" when running the periodic reclaim

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alvaro Lopez <1587356@xxxxxxxxxxxxxxxxxx>
Date: Tue, 31 May 2016 09:11:56 -0000
Reply-to: Bug 1587356 <1587356@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:


Nova sometimes raises an exception when running the periodic instance deleted,
see below. The error is quite spurious, and it is not happening always so I
cannot get the reason why this is happening. The periodic reclaim is trying to
delete the instance, but the DB is not returning the instance as it is marked
as deleted.

    ERROR nova.compute.manager Traceback (most recent call last):
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 447, in _object_dispatch
    ERROR nova.compute.manager     return getattr(target, method)(*args, **kwargs)
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
    ERROR nova.compute.manager     return fn(self, *args, **kwargs)
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 673, in save
    ERROR nova.compute.manager     columns_to_join=_expected_cols(expected_attrs))
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 765, in instance_update_and_get_original
    ERROR nova.compute.manager     expected=expected)
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper
    ERROR nova.compute.manager     return f(*args, **kwargs) 
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
    ERROR nova.compute.manager     ectxt.value = e.inner_exc 
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
    ERROR nova.compute.manager     return f(*args, **kwargs) 
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2478, in instance_update_and_get_original
    ERROR nova.compute.manager     session=session)
    ERROR nova.compute.manager
    ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1733, in _instance_get_by_uuid
    ERROR nova.compute.manager     raise exception.InstanceNotFound(instance_id=uuid)
    ERROR nova.compute.manager
    ERROR nova.compute.manager InstanceNotFound: Instance aceb4514-915e-4687-ac99-20f6b7d5ed6d could not be found.
    ERROR nova.compute.manager
    ERROR nova.compute.manager
    WARNING nova.compute.manager [req-e62c7e7c-0799-4f87-8f06-a46cf5663733 - - - - -] [instance: aceb4514-915e-4687-ac99-20f6b7d5ed6d] Periodic reclaim failed to delete instance: Instance aceb4514-915e-4687-ac99-20f6b7d5ed6d could not be found.

As I said, this is an spurious error, but from time to time we get a bunch of
these errors and we need to manually delete those instances.

** Affects: nova
     Importance: Undecided
     Assignee: Alvaro Lopez (aloga)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Alvaro Lopez (aloga)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1587356

Title:
  nova raises "InstanceNotFound" when running the periodic reclaim

Status in OpenStack Compute (nova):
  New

Bug description:
  
  Nova sometimes raises an exception when running the periodic instance deleted,
  see below. The error is quite spurious, and it is not happening always so I
  cannot get the reason why this is happening. The periodic reclaim is trying to
  delete the instance, but the DB is not returning the instance as it is marked
  as deleted.

      ERROR nova.compute.manager Traceback (most recent call last):
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 447, in _object_dispatch
      ERROR nova.compute.manager     return getattr(target, method)(*args, **kwargs)
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
      ERROR nova.compute.manager     return fn(self, *args, **kwargs)
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 673, in save
      ERROR nova.compute.manager     columns_to_join=_expected_cols(expected_attrs))
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 765, in instance_update_and_get_original
      ERROR nova.compute.manager     expected=expected)
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper
      ERROR nova.compute.manager     return f(*args, **kwargs) 
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
      ERROR nova.compute.manager     ectxt.value = e.inner_exc 
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
      ERROR nova.compute.manager     six.reraise(self.type_, self.value, self.tb)
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
      ERROR nova.compute.manager     return f(*args, **kwargs) 
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2478, in instance_update_and_get_original
      ERROR nova.compute.manager     session=session)
      ERROR nova.compute.manager
      ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1733, in _instance_get_by_uuid
      ERROR nova.compute.manager     raise exception.InstanceNotFound(instance_id=uuid)
      ERROR nova.compute.manager
      ERROR nova.compute.manager InstanceNotFound: Instance aceb4514-915e-4687-ac99-20f6b7d5ed6d could not be found.
      ERROR nova.compute.manager
      ERROR nova.compute.manager
      WARNING nova.compute.manager [req-e62c7e7c-0799-4f87-8f06-a46cf5663733 - - - - -] [instance: aceb4514-915e-4687-ac99-20f6b7d5ed6d] Periodic reclaim failed to delete instance: Instance aceb4514-915e-4687-ac99-20f6b7d5ed6d could not be found.

  As I said, this is an spurious error, but from time to time we get a bunch of
  these errors and we need to manually delete those instances.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1587356/+subscriptions