zeitgeist team mailing list archive

[Siegfried Gevatter <rainct@xxxxxxxxxx>]

** Description changed:

+ MOVE EVENTS
+ ============================================
  
- <RainCT> seiflotfy: the query updating current_uri on MOVE_EVENT should only change stuff with timestamp<move_event_timestamp
- <seiflotfy> RainCT, true
- <seiflotfy> good catch
- <RainCT> seiflotfy: there's also another ugly case
- <seiflotfy> RainCT, do tell
- <RainCT> seiflotfy: Imagine you insert event: 0. A, 1. A, 2. A, 3. B, 4. C, 5. A. Then with timestamp between events 3 and 4 you get A->T, so now you have "T, T, T, B, C, A"
- [...]
- <seiflotfy> i see the problem
- <seiflotfy> the last A should be a T
- <RainCT> no, the last A is fine
- <RainCT> because it is a new file with the same name
- [...]
- <seiflotfy> yeah ok
- <RainCT> seiflotfy: now you get A-L with timestamp between 1 and 2, so it should have "L, L, T, B, C, A", but since the current_uri of the first is already L you won't see it. for this it'd need to check the original URI instead of the current_uri
- <RainCT> are you with me so far?
- <seiflotfy> trying to
- <seiflotfy> RainCT, ok i cont get your last point
- <seiflotfy> A-L wont change the T
- <seiflotfy> because A has been move to T
- <seiflotfy> you can not move A again
- <seiflotfy> you need to move T
- <seiflotfy> thus its does not work
- <RainCT> yeah, but it should, because you're being told that it was moved before that
- <seiflotfy> unless you really want to you will have to look for all "MOVE_EVENTS" with A and figure out what A is now
- <seiflotfy> its doable
- <RainCT> so the move that happened later in time didn't affect those events, only the later ones
- <seiflotfy> RainCT, true
- <RainCT> the easy way to solve this is checking subj_id instead of subj_id_current
- <seiflotfy> RainCT, i think we should raise an exception
- <RainCT> but now when it gets really messed up is if there was even another move event before that
- <RainCT> which was already logged
- <seiflotfy> "You tried to move and event after it was used in a new location"
- <seiflotfy> RainCT, actually we also have the MOVE_EVENT logged
- <seiflotfy> you can then try to figrue out the patch of A
- <RainCT> yes, that's the solution
- <seiflotfy> the path
- <seiflotfy> RainCT, but i highly discourage that
- <RainCT> you can find the previos move event and set timestamp>previous_move_event.timestamp
- <seiflotfy> RainCT, exactly
- <seiflotfy> i am +- 0 on that tbh
- <seiflotfy> not sure
- <RainCT> ok, I don't dislike finding the previous timestamp
- <RainCT> i'll open a bug
+ PRESENTATION
+ 
+ By definition, Zeitgeist's events are immutable, and the subject meta-data
+ they contain is a snapshot of how a given resource was back when the event
+ happened.
+ 
+ To be useful, some way of linking event subjects to their physical
+ representation is needed. The primary identifier for doing this is the
+ subject's URI.
+ 
+ However, URIs, especially local ones, are transient and may change. To solve
+ this problem, a new field was added to subjects, and it is special in that
+ it isn't considered to be immutable. This is the `current_uri' field.
+ 
+ INITIAL IDEA
+ 
+ When a subject is inserted, its `current_uri' field is initially set to the
+ same value as its `uri' field. When Zeitgeist receives a MOVE_EVENT for that
+ file (with a coherent timestamp), the value of `current_uri' is updated to
+ its new file name.
+ 
+ The idea here is that this is done in a way that, if we deleted the
+ `current_uri' of all subjects and restored them looking at all MOVE_EVENTs
+ in the database, the result would be the same as before.
+ 
+ CURRENT IMPLEMENTATION
+ 
+ As of now, `current_uri' is initially set to the same value as `current_uri'.
+ Once a MOVE_EVENT is inserted, all events with a timestamp before that of the
+ move are updated.
+ 
+ However, after the point the MOVE_EVENT has been inserted, it is never
+ considered again. This is so for performance reasons, since the initial plan
+ would require pretty much "rebuilding the database".
+ 
+ PROBLEMS
+ 
+ There are numerous problems with this implementation, at least in theoretical
+ situations.
+ 
+ One problem is that of events coming in after the MOVE_EVENT (maybe because
+ the application is batching them). In this case they won't be updated.
+ 
+ We also have the opposite problem, a MOVE_EVENT coming in late after another
+ conflicting MOVE_EVENT happened. For instance, we have the following events:
+  > T5 a.txt, T10 a.txt, T15 a.txt
+ We receive a first MOVE_EVENT from a.txt to b.txt with timestamp T7. Now we
+ have (time / current_uri):
+  > T5 a.txt, T10 b.txt, T15 b.txt
+ Finally, we receive a further MOVE_EVENT from a.txt to c.txt with timestamp T0.
+ The result is:
+  > T5 c.txt, T10 b.txt, T15 b.txt
+ This is totally inconsistent; the correct result would have been:
+  > T5 c.txt, T10 c.txt, T15 b.txt
+ 
+ Further, even if implemented as described in the "initial idea" section, the
+ concept is flawed in that it may happen that events are inserted
+ retrospectively using already their updated URI. This could give rise to
+ further inconsistencies.
+ 
+ PROPOSAL
+ 
+ No clear way to avoid this problem is evident. Maybe the best idea is to
+ formalize the current behavior by documenting it and requesting that MOVE
+ and DELETE events be inserted near real time (for local files).
+ 
+ ADDITIONAL PROPOSAL
+ 
+ So far we haven't taken resource deletions into account at all. However,
+ those also affect the URI of a resource, in that it ceases to exist (and
+ may be subsequently reused for an unrelated resource).
+ 
+ For this reason, I propose that DELETE_EVENTs also update `current_uri'. In
+ particular, they should change said URI to "" (empty).

-- 
You received this bug notification because you are a member of Zeitgeist
Framework Team, which is subscribed to Zeitgeist Framework.
https://bugs.launchpad.net/bugs/778140

Title:
  Time warp problem in MOVE_EVENT handling

Status in Zeitgeist Framework:
  Triaged

Bug description:
  MOVE EVENTS
  ============================================

  PRESENTATION

  By definition, Zeitgeist's events are immutable, and the subject meta-data
  they contain is a snapshot of how a given resource was back when the event
  happened.

  To be useful, some way of linking event subjects to their physical
  representation is needed. The primary identifier for doing this is the
  subject's URI.

  However, URIs, especially local ones, are transient and may change. To solve
  this problem, a new field was added to subjects, and it is special in that
  it isn't considered to be immutable. This is the `current_uri' field.

  INITIAL IDEA

  When a subject is inserted, its `current_uri' field is initially set to the
  same value as its `uri' field. When Zeitgeist receives a MOVE_EVENT for that
  file (with a coherent timestamp), the value of `current_uri' is updated to
  its new file name.

  The idea here is that this is done in a way that, if we deleted the
  `current_uri' of all subjects and restored them looking at all MOVE_EVENTs
  in the database, the result would be the same as before.

  CURRENT IMPLEMENTATION

  As of now, `current_uri' is initially set to the same value as `current_uri'.
  Once a MOVE_EVENT is inserted, all events with a timestamp before that of the
  move are updated.

  However, after the point the MOVE_EVENT has been inserted, it is never
  considered again. This is so for performance reasons, since the initial plan
  would require pretty much "rebuilding the database".

  PROBLEMS

  There are numerous problems with this implementation, at least in theoretical
  situations.

  One problem is that of events coming in after the MOVE_EVENT (maybe because
  the application is batching them). In this case they won't be updated.

  We also have the opposite problem, a MOVE_EVENT coming in late after another
  conflicting MOVE_EVENT happened. For instance, we have the following events:
   > T5 a.txt, T10 a.txt, T15 a.txt
  We receive a first MOVE_EVENT from a.txt to b.txt with timestamp T7. Now we
  have (time / current_uri):
   > T5 a.txt, T10 b.txt, T15 b.txt
  Finally, we receive a further MOVE_EVENT from a.txt to c.txt with timestamp T0.
  The result is:
   > T5 c.txt, T10 b.txt, T15 b.txt
  This is totally inconsistent; the correct result would have been:
   > T5 c.txt, T10 c.txt, T15 b.txt

  Further, even if implemented as described in the "initial idea" section, the
  concept is flawed in that it may happen that events are inserted
  retrospectively using already their updated URI. This could give rise to
  further inconsistencies.

  PROPOSAL

  No clear way to avoid this problem is evident. Maybe the best idea is to
  formalize the current behavior by documenting it and requesting that MOVE
  and DELETE events be inserted near real time (for local files).

  ADDITIONAL PROPOSAL

  So far we haven't taken resource deletions into account at all. However,
  those also affect the URI of a resource, in that it ceases to exist (and
  may be subsequently reused for an unrelated resource).

  For this reason, I propose that DELETE_EVENTs also update `current_uri'. In
  particular, they should change said URI to "" (empty).

To manage notifications about this bug go to:
https://bugs.launchpad.net/zeitgeist/+bug/778140/+subscriptions