launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

I do hope these emails are useful or at least entertaining for folk :)

So, I'm signing off for the night, and I've only made a little
progress today, but I thought I'd share it, hoping its useful - and
adding to my backlog of things to find good places on the wiki for.

I (hopefully) helped registry fix a timeout - DistroSeries.packagings
was doing listification, and that meant that a trivial len() in
browser code turned into a very slow operation. This isn't a golden
rule of course - sometimes lists *are* better, but in this case I'm
pretty sure that DecoratedResultSet (which is mind bendingly easy to
use) will help.
https://bugs.edge.launchpad.net/launchpad-registry/+bug/612358. I seem
to have set the cat amongst the pidgeons with rc-approving it after
Curtis reviewed it, because I wrote it : I'm going to dig into that in
a separate thread.

The same basic thing was used in the previous branch I did to help
/participants - and thats approved now pending a) getting out of
freeze, b) me fixing test fallout that happens because production
cleans caches out that the test suite doesn't.

While I was there I noted a few places that have the idiom
result = some_storm_result_set
return [
  foo for
  foo, bar,baz, quxx in result]

This is the basic idiom that was *causing* the two timeouts I mention
above. I think it should be considered a red flag for reviewers and
devs coding - is that the current feeling about it? The reason being
that even if the first use of a query is for (say) a batch process
that examines every row, someone *will* come along and want to just
get the first, or any, of the items, and listification really hurts
then. DecoratedResultSet can do this trivially:
result = some_storm_result_set
return DecoratedResultSet(result, lambda row:row[0])

I was looking at working on
https://bugs.edge.launchpad.net/malone/+bug/607960, but I realised
that its not on the radar anymore - its slow but not causing timeouts
for now. I'd like to fix it but it needs some plumbing that is non
trivial and because its not affecting use, I just  gardened it and
took it out of the timeout queue. I haven't reassigned the priority -
I'm not sure what it should be now ;). I also split out a dedicated
bug for one of the sub-discussions that took place there.
https://bugs.edge.launchpad.net/launchpad-foundations/+bug/615685

I helped wgrant out with some explain analyzes - it seems to me it
might be nice - handwaving - to permit explain-analyze calls by any
developer, but perhaps thats just insanity at this hour of the night
kicking in:)

Lastly I've been looking at the timeouts on +packaging
https://bugs.edge.launchpad.net/soyuz/+bug/607934, but I haven't got
very far - I've got a test that shows the page scales at 4 queries per
package, which leads to 200+ queries on a 50 package batch, but the
code doing the work is full of indirections via getUtility that will
need removing to consolidate the work - so I'm going to put the
patch-as-it-stands at the bottom of this email and probably come back
to it next tuesday.

Happy performance day-ly, yours
-Rob