launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> In a previous life, the context data that I've used for this is a timestamp, 
> and it worked very well in pretty much all cases I came across.  The client 
> application simply provides the same timestamp to a query/api call from the 
> last item it processed, and the data continues to flow from where it left off.  
> This ticked all the boxes for data integrity and polling or streaming usage.

Timestamps are an approximation of sequence, because sometimes there are
multiple rows with the same timestamp.  This is not unlikely, because
the multiple rows may created as part of a single transaction.

Because there's room for ambiguity, a process such as you describe could
1. refuse to stop while the next row to be processed has the same
   timestamp as the current row.
2. stop in the middle and when it starts again, skip the remainder of
   items with the same timestamp.
3. stop in the middle and when it starts again, start from the first
   item with that timestamp.
4. ?

1. could work, if it's not essential that we stop immediately.
2. is usually undesirable, but can sometimes be fixed up by a second
   cron job that detects that the work still needs to be done.
3. could work, if running the operation twice for a given item doesn't
   do any harm.  However, we could get stopped again, and again, and
   again, and never finish running the operation on all rows.

As I said, timestamps are an approximation of sequence, but we have
genuine sequences for pretty much every table: integer ID columns.  If
you order the operations by database ID rather than by timestamp, then
you can record the last ID completed, and there is no room for
ambiguity.  So I think it's simpler to use database IDs rather than
timestamps.

Your idea reminds me of two things:
1. DBLoopTuner
2. "micro-jobs" from
https://dev.launchpad.net/Foundations/NewTaskSystem/Requirements

Perhaps those could also provide inspiration?

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0jSc4ACgkQ0F+nu1YWqI2I1QCdHUgcusgJYmLrOImz3RJJ790X
Y64An3YSo/23mCak0/MBw/NPhEUZlZH0
=zu/n
-----END PGP SIGNATURE-----