yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Public bug reported:

This change https://review.openstack.org/#/c/163515/ added a warning
when the @reverts_task_state decorator in the compute manager fails
rather than just pass, because we were getting KeyErrors and never
noticing them which broke the decorator.

However, now we're tracing on InstanceNotFound which is a normal case if
we're deleting the instance after a failure (tempest will delete the
instance immediately after failures when tearing down a test):

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHJldmVydCB0YXNrIHN0YXRlIGZvciBpbnN0YW5jZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjYxNzA3MDE2OTV9

http://logs.openstack.org/98/163798/1/check/check-tempest-dsvm-postgres-
full/6eff665/logs/screen-n-cpu.txt.gz#_2015-03-12_13_11_36_304

2015-03-12 13:11:36.304 WARNING nova.compute.manager [req-a5f3b37e-19e9-4e1d-9be7-bbb9a8e7f4c1 DeleteServersTestJSON-706956764 DeleteServersTestJSON-535578435] [instance: 6de2ad51-3155-4538-830d-f02de39b4be3] Failed to revert task state for instance. Error: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could not be found.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
    return func(*args, **kwargs)

  File "/opt/stack/new/nova/nova/conductor/manager.py", line 134, in instance_update
    columns_to_join=['system_metadata'])

  File "/opt/stack/new/nova/nova/db/api.py", line 774, in instance_update_and_get_original
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2395, in instance_update_and_get_original
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 181, in wrapped
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2434, in _instance_update
    columns_to_join=columns_to_join)

  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1670, in _instance_get_by_uuid
    raise exception.InstanceNotFound(instance_id=uuid)

InstanceNotFound: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could
not be found.

** Affects: nova
     Importance: Medium
     Assignee: Matt Riedemann (mriedem)
         Status: In Progress


** Tags: compute icehouse-backport-potential juno-backport-potential

** Tags removed: junot
** Tags added: icehouse-backport-potential juno-backport-potential

** Changed in: nova
     Assignee: (unassigned) => Matt Riedemann (mriedem)

** Changed in: nova
   Importance: Undecided => Medium

** Changed in: nova
       Status: New => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1431404

Title:
  Don't trace when @reverts_task_state fails on InstanceNotFound

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  This change https://review.openstack.org/#/c/163515/ added a warning
  when the @reverts_task_state decorator in the compute manager fails
  rather than just pass, because we were getting KeyErrors and never
  noticing them which broke the decorator.

  However, now we're tracing on InstanceNotFound which is a normal case
  if we're deleting the instance after a failure (tempest will delete
  the instance immediately after failures when tearing down a test):

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHJldmVydCB0YXNrIHN0YXRlIGZvciBpbnN0YW5jZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjYxNzA3MDE2OTV9

  http://logs.openstack.org/98/163798/1/check/check-tempest-dsvm-
  postgres-
  full/6eff665/logs/screen-n-cpu.txt.gz#_2015-03-12_13_11_36_304

  2015-03-12 13:11:36.304 WARNING nova.compute.manager [req-a5f3b37e-19e9-4e1d-9be7-bbb9a8e7f4c1 DeleteServersTestJSON-706956764 DeleteServersTestJSON-535578435] [instance: 6de2ad51-3155-4538-830d-f02de39b4be3] Failed to revert task state for instance. Error: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could not be found.
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
      return func(*args, **kwargs)

    File "/opt/stack/new/nova/nova/conductor/manager.py", line 134, in instance_update
      columns_to_join=['system_metadata'])

    File "/opt/stack/new/nova/nova/db/api.py", line 774, in instance_update_and_get_original
      columns_to_join=columns_to_join)

    File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 143, in wrapper
      return f(*args, **kwargs)

    File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2395, in instance_update_and_get_original
      columns_to_join=columns_to_join)

    File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 181, in wrapped
      return f(*args, **kwargs)

    File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2434, in _instance_update
      columns_to_join=columns_to_join)

    File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1670, in _instance_get_by_uuid
      raise exception.InstanceNotFound(instance_id=uuid)

  InstanceNotFound: Instance 6de2ad51-3155-4538-830d-f02de39b4be3 could
  not be found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1431404/+subscriptions