yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Added Cinder since there are at least two periodic tasks in Cinder that
don't have specific spacing set.

** Also affects: cinder
   Importance: Undecided
       Status: New

** Also affects: neutron
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1319232

Title:
  Periodic tasks run too frequently

Status in Cinder:
  New
Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  Fix Committed
Status in Oslo - a Library of Common OpenStack Code:
  Fix Committed

Bug description:
  Each periodic task can have a "spacing", which defines the minimum
  amount of time between executions of that task.  For example, a task
  with periodic_spacing=120 would execute no more often than once every
  2 minutes.  Tasks that do not define an explicit spacing will be run
  every time the periodic task processor runs.  This is commonly loosely
  interpreted as "every 60 seconds", but in reality it's more
  complicated than that.

  As a result of these "complications", we can actually end up running
  these tasks more frequently -- I've regularly observed them running
  every 20-30 seconds, and in several cases I've seen a task running
  just 1-2 seconds after it previously ran.  This consumes extra
  resources (CPU, database access, etc) without providing any real
  value.

  The reason for these extra runs has to do with how the periodic task
  processor is implemented.  When there are multiple tasks with a
  defined spacing, they can get somewhat staggered and force the
  periodic task processor to run additional iterations.  Since tasks
  with no spacing run every time the periodic task processor runs, they
  get run more frequently than one would expect.

  
  My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds).  The code change is simple -- in nova/openstack/common/periodic_task.py, change this code:

          # A periodic spacing of zero indicates that this task should
          # be run every pass
          if task._periodic_spacing == 0:
              task._periodic_spacing = None

  to:
          # A periodic spacing of zero indicates that this task should
          # be run at the default interval
          if task._periodic_spacing == 0:
              task._periodic_spacing = DEFAULT_INTERVAL

  The actual runtime task processing code doesn't change -- this fix is
  basically the equivalent of finding every @periodic_task that doesn't
  have an explicit spacing, and setting spacing=60.  So it's very low
  risk.  Some may argue that this change in behavior could cause some
  task to behave differently than it used to.  However, there was never
  any guarantee that the task would run more often than every 60
  seconds, and in many cases the tasks may already run less frequently
  than that (due to other long-running tasks).  So this change should
  not introduce any new issues related to the timing of task execution;
  it would only serve to make the timing more regular.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1319232/+subscriptions