yahoo-eng-team team mailing list archive

[Alex Xu <xuhj@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When execute some action on instance, instance will update task state
first. For example:

        instance.task_state = task_states.SHELVING
        instance.save(expected_task_state=[None])

When update task_state, it will check the expected_task_state first.
But there is race problem when check the expected task state.

def _instance_update(context, instance_uuid, values, copy_old_instance=False,
                     columns_to_join=None):
    session = get_session()

    if not uuidutils.is_uuid_like(instance_uuid):
        raise exception.InvalidUUID(instance_uuid)

    with session.begin():
        instance_ref = _instance_get_by_uuid(context, instance_uuid,
                                             session=session,
                                             columns_to_join=columns_to_join)
        if "expected_task_state" in values:
            # it is not a db column so always pop out
            expected = values.pop("expected_task_state")
            if not isinstance(expected, (tuple, list, set)):
                expected = (expected,)

<!-----
# If there are two request concurrence for this transaction, second request already modify the instance's task state, but the first request's still get the old state.
# we need use lock for get instance from db
----!>

            actual_state = instance_ref["task_state"]
            if actual_state not in expected:
                if actual_state == task_states.DELETING:
                    raise exception.UnexpectedDeletingTaskStateError(
                            actual=actual_state, expected=expected)
                else:
                    raise exception.UnexpectedTaskStateError(
                            actual=actual_state, expected=expected)

** Affects: nova
     Importance: Undecided
     Assignee: Alex Xu (xuhj)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Alex Xu (xuhj)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1291246

Title:
  race problem when checking expected_task_state in instance updating

Status in OpenStack Compute (Nova):
  New

Bug description:
  When execute some action on instance, instance will update task state
  first. For example:

          instance.task_state = task_states.SHELVING
          instance.save(expected_task_state=[None])

  When update task_state, it will check the expected_task_state first.
  But there is race problem when check the expected task state.

  def _instance_update(context, instance_uuid, values, copy_old_instance=False,
                       columns_to_join=None):
      session = get_session()

      if not uuidutils.is_uuid_like(instance_uuid):
          raise exception.InvalidUUID(instance_uuid)

      with session.begin():
          instance_ref = _instance_get_by_uuid(context, instance_uuid,
                                               session=session,
                                               columns_to_join=columns_to_join)
          if "expected_task_state" in values:
              # it is not a db column so always pop out
              expected = values.pop("expected_task_state")
              if not isinstance(expected, (tuple, list, set)):
                  expected = (expected,)

  <!-----
  # If there are two request concurrence for this transaction, second request already modify the instance's task state, but the first request's still get the old state.
  # we need use lock for get instance from db
  ----!>

              actual_state = instance_ref["task_state"]
              if actual_state not in expected:
                  if actual_state == task_states.DELETING:
                      raise exception.UnexpectedDeletingTaskStateError(
                              actual=actual_state, expected=expected)
                  else:
                      raise exception.UnexpectedTaskStateError(
                              actual=actual_state, expected=expected)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1291246/+subscriptions