launchpad-reviewers team mailing list archive

[Michael Nelson <michael.nelson@xxxxxxxxxxxxx>]

The proposal to merge lp:~michael.nelson/launchpad/982938-dont-load-all-tokens-for-all-ppas-into-memory into lp:launchpad has been updated.

Description changed to:

Summary
=======
This branch aims to reduce the load/memory consumption on germanium while generating htpasswd files.

Proposed fix
============

Use a generator to return a storm result set of tokens for each PPA, rather than
loading all tokens for all PPAs into a dict.

It will result in a larger query count. jml is working on a script that will generate data to be able to verify whether this branch does indeed help.

Pre-implementation notes
========================
Referring to https://bugs.launchpad.net/launchpad/+bug/982938/comments/6
12:27 < noodles> jml: if you think the first of those points (in my comment) is very likely to be causing performance issues, I could start coding that straight away...
12:27 < noodles> (given the numbers achuni mentioned above, it would seem likely, but only the measurements will tell).
12:27 < jml> as you say, only measurements will tell :)
12:28 < jml> noodles: I reckon code it up straight away. :)

Implementation details
======================
Sheesh - this is the quick cover letter? :P

LOC Rationale
=============
Yay - 12 lines of credit.

Tests
=====
./bin/test -vvct TestPPAHtaccessTokenGeneration
./bin/test -vvct TestHtpasswdGeneration

Demo and Q/A
============
If this script runs on staging, we could QA by comparing output before and after running it.

Lint
====
Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/scripts/generate_ppa_htaccess.py

For more details, see:
https://code.launchpad.net/~michael.nelson/launchpad/982938-dont-load-all-tokens-for-all-ppas-into-memory/+merge/107980
-- 
https://code.launchpad.net/~michael.nelson/launchpad/982938-dont-load-all-tokens-for-all-ppas-into-memory/+merge/107980
Your team Launchpad code reviewers is subscribed to branch lp:launchpad.