launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Max Kanat-Alexander wrote:
> On 11/19/2009 07:50 PM, Michael Hudson wrote:
>>> 	So the question for this issue is--what level would you like me to
>>> address it on? If you'd like me to simply work on the revision graph
>>> issue, I could do that within the current architecture of loggerhead and
>>> devise a fix. Probably the simplest would be to just place a mutex
>>> around building a revision graph for any one branch.
>> That's probably a good fix for loggerhead, but maybe not sufficient for
>> Launchpad.
> 
> 	Okay. Well, maybe to shore up the problem a little, we could do one and
> then the other. The mutex thing would be pretty quick, the scaling stuff
> would be a more significant amount of time.
> 
>> Another team at Canonical tried spawning and had to give up and go back
>> to paste.  So let's learn from their misfortune :)
> 
> 	Fair enough. :-) It was mostly just a suggestion with no research--the
> core idea being "let's scale better". :-)
> 
>> Speaking more generally, the problem is the revision cache -- can we
>> make it go away, or at least handle it better?
> 
> 	It would indeed seem to me like the things that loggerhead does are
> largely identical to the things that normal bzr users need to do, and
> that using bzrlib directly should be OK. Though perhaps on older
> repository formats, less so.

Oh man yes.  But I don't think that's realistic currently, if only
because of revision numbering.  The other reasons loggerhead crawls over
most of the revision graph on every page are probably avoidable (and hey
if you can figure out how, that would be great), but afaict there's no
solution in sight for off-mainline revision numbering.

>> Y'see, one of the problems with a central graph store is that graphs are
>> big, and any central store implies IPC which implies serialization, and
>> serializing and deserializing something as big as Launchpad's revision
>> graph cache is annoyingly slow.  So one idea would be to have this
>> central store not serve up entire graphs, but instead be able to answer
>> the questions above. 
> 
> 	One would think that bzr itself should be efficient at answering those
> questions, though, no? Except perhaps the merge_point_list. The stuff
> needed to show RevisionLogUI, for example, is identical to what's
> required to show "bzr log", yes?

ChangeLogUI corresponds most closely to log, but yes.  However, if you
time how long it takes for 'bzr log --include-merges' to start producing
output, you'll see that it's about 0.7s longer than how long it takes
'bzr log' to start producing output -- the difference is mostly building
the revid->revno map.  For CLI usage, this isn't too bad, for a webapp
it is.

> Similarly, RevisionUI is "bzr diff"
> plus a little log info. Almost every page corresponds to a common action
> taken by a normal bzr user.
> 
>> I'm not sure what the "quick patch" would be -- the mutex around
>> revision graph cache building?
> 
> 	Yeah.

OK, well that sounds worth doing in any case.

Cheers,
mwh