yahoo-eng-team team mailing list archive

[Diana Clarke <diana.joan.clarke@xxxxxxxxx>]

Public bug reported:

Heartbeats stop working when you mess with the system time. If a
monotonic clock were used, they would continue to work when the system
time was changed.

Steps to reproduce:

1. List the nova services ('nova-manage service list'). Note that the
'State' for each services is a happy face ':-)'.

2. Move the time ahead (for example 2 hours in the future), and then
list the nova services again. Note that heartbeats continue to work and
use the future time (see 'Updated_At').

3. Revert back to the actual time, and list the nova services again.
Note that all heartbeats stop, and have a 'State' of 'XXX'.

4. The heartbeats will start again in 2 hours when the actual time
catches up to the future time, or if you restart the services.

Here's example output demonstrating the issue:

    http://paste.openstack.org/show/477404/

See bug #1450438 for more context:

    https://bugs.launchpad.net/oslo.service/+bug/1450438

Long story short: looping call is using the built-in time rather than a
monotonic clock for sleeps.

https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122

Oslo Service: version 0.11
Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: oslo

** Description changed:

  Heartbeats stop working when you mess with the system time. If a
  monotonic clock were used, they would continue to work when the system
  time was changed.
  
  Steps to reproduce:
  
- * List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
- * Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
- * Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
- * The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
+ 1. List the nova services ('nova-manage service list'). Note that the
+ 'State' for each services is a happy face ':-)'.
+ 
+ 2. Move the time ahead (for example 2 hours in the future), and then
+ list the nova services again. Note that heartbeats continue to work and
+ use the future time (see 'Updated_At').
+ 
+ 3. Revert back to the actual time, and list the nova services again.
+ Note that all heartbeats stop, and have a 'State' of 'XXX'.
+ 
+ 4. The heartbeats will start again in 2 hours when the actual time
+ catches up to the future time, or if you restart the services.
  
  Here's example output demonstrating the issue:
  
-     http://paste.openstack.org/show/477404/
+     http://paste.openstack.org/show/477404/
  
  See bug #1450438 for more context:
  
-     https://bugs.launchpad.net/oslo.service/+bug/1450438
+     https://bugs.launchpad.net/oslo.service/+bug/1450438
  
  Long story short: looping call is using the built-in time rather than a
  monotonic clock for sleeps.
  
  https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
  
  Oslo Service: version 0.11
  Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)

** Description changed:

  Heartbeats stop working when you mess with the system time. If a
  monotonic clock were used, they would continue to work when the system
  time was changed.
  
  Steps to reproduce:
  
  1. List the nova services ('nova-manage service list'). Note that the
  'State' for each services is a happy face ':-)'.
  
  2. Move the time ahead (for example 2 hours in the future), and then
  list the nova services again. Note that heartbeats continue to work and
  use the future time (see 'Updated_At').
  
  3. Revert back to the actual time, and list the nova services again.
  Note that all heartbeats stop, and have a 'State' of 'XXX'.
  
  4. The heartbeats will start again in 2 hours when the actual time
  catches up to the future time, or if you restart the services.
  
  Here's example output demonstrating the issue:
  
      http://paste.openstack.org/show/477404/
  
  See bug #1450438 for more context:
  
      https://bugs.launchpad.net/oslo.service/+bug/1450438
  
  Long story short: looping call is using the built-in time rather than a
  monotonic clock for sleeps.
  
  https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122
  
  Oslo Service: version 0.11
- Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)
+ Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1510234

Title:
  Heartbeats stop when time is changed

Status in OpenStack Compute (nova):
  New

Bug description:
  Heartbeats stop working when you mess with the system time. If a
  monotonic clock were used, they would continue to work when the system
  time was changed.

  Steps to reproduce:

  1. List the nova services ('nova-manage service list'). Note that the
  'State' for each services is a happy face ':-)'.

  2. Move the time ahead (for example 2 hours in the future), and then
  list the nova services again. Note that heartbeats continue to work
  and use the future time (see 'Updated_At').

  3. Revert back to the actual time, and list the nova services again.
  Note that all heartbeats stop, and have a 'State' of 'XXX'.

  4. The heartbeats will start again in 2 hours when the actual time
  catches up to the future time, or if you restart the services.

  Here's example output demonstrating the issue:

      http://paste.openstack.org/show/477404/

  See bug #1450438 for more context:

      https://bugs.launchpad.net/oslo.service/+bug/1450438

  Long story short: looping call is using the built-in time rather than
  a  monotonic clock for sleeps.

  https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122

  Oslo Service: version 0.11
  Nova: master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1510234/+subscriptions