launchpad-reviewers team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxx>]

Turns out that all the memory was taken by the url_times cache. In 300k requests, we have nearly 1k different URLS! All of this to generate the top 200. So I now limit the size of that cache to 50 times the top_n we are looking for, and culling it by 10% everything we reach the limit.

I've tried various size and cull ratio, and this seems a good size/time/accuracy compromise. Generating the report now takes a constant 137M and the top200 report is identical to one generated with all of them.

It actually reduced the time to generate the report also by 9 seconds.

I also now persist the RequestTimes data structure to disk. This will allow us to generate aggregate report very fast now. And that structure takes ~1.8M of disk space once compressed.
-- 
https://code.launchpad.net/~flacoste/launchpad/ppr-constant-memory/+merge/39666
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~flacoste/launchpad/ppr-constant-memory into lp:launchpad/devel.