← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #33277

[Bug 1460350] [NEW] Cells: Race deleting instance can lead to instances "undeleted" at the top

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Seen in check-tempest-dsvm-cells job failure, example trace from [1]:

Traceback (most recent call last):
  File "tempest/api/compute/servers/test_list_servers_negative.py", line 153, in test_list_servers_detail_server_is_deleted
    self.assertEqual([], actual)
  File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = []
actual    = [{u'OS-DCF:diskConfig': u'MANUAL',
  u'OS-EXT-AZ:availability_zone': u'nova',
  u'OS-EXT-STS:power_state': 0,
  u'OS-EXT-STS:task_state': None,
  u'OS-EXT-STS:vm_state': u'deleted',
  u'OS-SRV-USG:launched_at': None,
  u'OS-SRV-USG:terminated_at': u'2015-05-17T15:46:15.000000',
  u'accessIPv4': u'',
  u'accessIPv6': u'',
  u'addresses': {},
  u'config_drive': u'',
  u'created': u'2015-05-17T15:46:15Z',
  u'flavor': {u'id': u'42',
              u'links': [{u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/flavors/42',
                          u'rel': u'bookmark'}]},
  u'hostId': u'',
  u'id': u'45b1decf-8f52-4075-8869-acb9f48de159',
  u'image': {u'id': u'990c6a37-da73-4a74-be3d-eff98dcf7727',
             u'links': [{u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/images/990c6a37-da73-4a74-be3d-eff98dcf7727',
                         u'rel': u'bookmark'}]},
  u'key_name': None,
  u'links': [{u'href': u'http://127.0.0.1:8774/v2/82eeb74985844a9daa71b162f663e981/servers/45b1decf-8f52-4075-8869-acb9f48de159',
              u'rel': u'self'},
             {u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/servers/45b1decf-8f52-4075-8869-acb9f48de159',
              u'rel': u'bookmark'}],
  u'metadata': {},
  u'name': u'ListServersNegativeTestJSON-instance-1205034409',
  u'os-extended-volumes:volumes_attached': [],
  u'status': u'DELETED',
  u'tenant_id': u'82eeb74985844a9daa71b162f663e981',
  u'updated': u'2015-05-17T15:46:15Z',
  u'user_id': u'aac5bd38fc264166b9b365426557b4d2'}]

The test creates an instance and immediately deletes it before it's
scheduled. After the delete has happened at the top via local delete,
updates from the child cells can arrive and "undelete" the instance.
This is possible because the code in nova/cells/messaging.py does
read_deleted='yes' and db.instance_update() will update all fields
provided (unlike objects). I also tried removing the local delete logic
in nova/compute/cells_api.py and it didn't help -- the destroy in the
child will trigger a instance_destroy_at_top() but it's still possible
for the instance.save() update to occur after the destroy, resulting
again in "undeleted" instance.

This issue should go away when instance_update_at_top() is converted to
use objects, as the "deleted" etc fields won't ever be in what_changed.

[1] http://logs.openstack.org/09/183909/2/check/check-tempest-dsvm-
cells/9799bb0

** Affects: nova
     Importance: Low
     Assignee: melanie witt (melwitt)
         Status: Triaged


** Tags: cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1460350

Title:
  Cells: Race deleting instance can lead to instances "undeleted" at the
  top

Status in OpenStack Compute (Nova):
  Triaged

Bug description:
  Seen in check-tempest-dsvm-cells job failure, example trace from [1]:

  Traceback (most recent call last):
    File "tempest/api/compute/servers/test_list_servers_negative.py", line 153, in test_list_servers_detail_server_is_deleted
      self.assertEqual([], actual)
    File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
      self.assertThat(observed, matcher, message)
    File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: !=:
  reference = []
  actual    = [{u'OS-DCF:diskConfig': u'MANUAL',
    u'OS-EXT-AZ:availability_zone': u'nova',
    u'OS-EXT-STS:power_state': 0,
    u'OS-EXT-STS:task_state': None,
    u'OS-EXT-STS:vm_state': u'deleted',
    u'OS-SRV-USG:launched_at': None,
    u'OS-SRV-USG:terminated_at': u'2015-05-17T15:46:15.000000',
    u'accessIPv4': u'',
    u'accessIPv6': u'',
    u'addresses': {},
    u'config_drive': u'',
    u'created': u'2015-05-17T15:46:15Z',
    u'flavor': {u'id': u'42',
                u'links': [{u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/flavors/42',
                            u'rel': u'bookmark'}]},
    u'hostId': u'',
    u'id': u'45b1decf-8f52-4075-8869-acb9f48de159',
    u'image': {u'id': u'990c6a37-da73-4a74-be3d-eff98dcf7727',
               u'links': [{u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/images/990c6a37-da73-4a74-be3d-eff98dcf7727',
                           u'rel': u'bookmark'}]},
    u'key_name': None,
    u'links': [{u'href': u'http://127.0.0.1:8774/v2/82eeb74985844a9daa71b162f663e981/servers/45b1decf-8f52-4075-8869-acb9f48de159',
                u'rel': u'self'},
               {u'href': u'http://127.0.0.1:8774/82eeb74985844a9daa71b162f663e981/servers/45b1decf-8f52-4075-8869-acb9f48de159',
                u'rel': u'bookmark'}],
    u'metadata': {},
    u'name': u'ListServersNegativeTestJSON-instance-1205034409',
    u'os-extended-volumes:volumes_attached': [],
    u'status': u'DELETED',
    u'tenant_id': u'82eeb74985844a9daa71b162f663e981',
    u'updated': u'2015-05-17T15:46:15Z',
    u'user_id': u'aac5bd38fc264166b9b365426557b4d2'}]

  The test creates an instance and immediately deletes it before it's
  scheduled. After the delete has happened at the top via local delete,
  updates from the child cells can arrive and "undelete" the instance.
  This is possible because the code in nova/cells/messaging.py does
  read_deleted='yes' and db.instance_update() will update all fields
  provided (unlike objects). I also tried removing the local delete
  logic in nova/compute/cells_api.py and it didn't help -- the destroy
  in the child will trigger a instance_destroy_at_top() but it's still
  possible for the instance.save() update to occur after the destroy,
  resulting again in "undeleted" instance.

  This issue should go away when instance_update_at_top() is converted
  to use objects, as the "deleted" etc fields won't ever be in
  what_changed.

  [1] http://logs.openstack.org/09/183909/2/check/check-tempest-dsvm-
  cells/9799bb0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1460350/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next