yahoo-eng-team team mailing list archive

[Jeremy Arnold <arnoldje@xxxxxxxxxx>]

Public bug reported:

Each periodic task can have a "spacing", which defines the minimum
amount of time between executions of that task.  For example, a task
with periodic_spacing=120 would execute no more often than once every 2
minutes.  Tasks that do not define an explicit spacing will be run every
time the periodic task processor runs.  This is commonly loosely
interpreted as "every 60 seconds", but in reality it's more complicated
than that.

As a result of these "complications", we can actually end up running
these tasks more frequently -- I've regularly observed them running
every 20-30 seconds, and in several cases I've seen a task running just
1-2 seconds after it previously ran.  This consumes extra resources
(CPU, database access, etc) without providing any real value.

The reason for these extra runs has to do with how the periodic task
processor is implemented.  When there are multiple tasks with a defined
spacing, they can get somewhat staggered and force the periodic task
processor to run additional iterations.  Since tasks with no spacing run
every time the periodic task processor runs, they get run more
frequently than one would expect.


My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds).  The code change is simple -- in nova/openstack/common/periodic_task.py, change this code:

        # A periodic spacing of zero indicates that this task should
        # be run every pass
        if task._periodic_spacing == 0:
            task._periodic_spacing = None

to:
        # A periodic spacing of zero indicates that this task should
        # be run at the default interval
        if task._periodic_spacing == 0:
            task._periodic_spacing = DEFAULT_INTERVAL

The actual runtime task processing code doesn't change -- this fix is
basically the equivalent of finding every @periodic_task that doesn't
have an explicit spacing, and setting spacing=60.  So it's very low
risk.  Some may argue that this change in behavior could cause some task
to behave differently than it used to.  However, there was never any
guarantee that the task would run more often than every 60 seconds, and
in many cases the tasks may already run less frequently than that (due
to other long-running tasks).  So this change should not introduce any
new issues related to the timing of task execution; it would only serve
to make the timing more regular.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1319232

Title:
  Periodic tasks run too frequently

Status in OpenStack Compute (Nova):
  New

Bug description:
  Each periodic task can have a "spacing", which defines the minimum
  amount of time between executions of that task.  For example, a task
  with periodic_spacing=120 would execute no more often than once every
  2 minutes.  Tasks that do not define an explicit spacing will be run
  every time the periodic task processor runs.  This is commonly loosely
  interpreted as "every 60 seconds", but in reality it's more
  complicated than that.

  As a result of these "complications", we can actually end up running
  these tasks more frequently -- I've regularly observed them running
  every 20-30 seconds, and in several cases I've seen a task running
  just 1-2 seconds after it previously ran.  This consumes extra
  resources (CPU, database access, etc) without providing any real
  value.

  The reason for these extra runs has to do with how the periodic task
  processor is implemented.  When there are multiple tasks with a
  defined spacing, they can get somewhat staggered and force the
  periodic task processor to run additional iterations.  Since tasks
  with no spacing run every time the periodic task processor runs, they
  get run more frequently than one would expect.

  
  My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds).  The code change is simple -- in nova/openstack/common/periodic_task.py, change this code:

          # A periodic spacing of zero indicates that this task should
          # be run every pass
          if task._periodic_spacing == 0:
              task._periodic_spacing = None

  to:
          # A periodic spacing of zero indicates that this task should
          # be run at the default interval
          if task._periodic_spacing == 0:
              task._periodic_spacing = DEFAULT_INTERVAL

  The actual runtime task processing code doesn't change -- this fix is
  basically the equivalent of finding every @periodic_task that doesn't
  have an explicit spacing, and setting spacing=60.  So it's very low
  risk.  Some may argue that this change in behavior could cause some
  task to behave differently than it used to.  However, there was never
  any guarantee that the task would run more often than every 60
  seconds, and in many cases the tasks may already run less frequently
  than that (due to other long-running tasks).  So this change should
  not introduce any new issues related to the timing of task execution;
  it would only serve to make the timing more regular.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1319232/+subscriptions