launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Tue, Nov 29, 2011 at 5:59 AM, Gavin Panella
<gavin.panella@xxxxxxxxxxxxx> wrote:
> Cursor arraysize is the number of rows that psycopg2 fetches each time
> fetchmany() is called. Storm's Result.__iter__() rests upon that, and
> ResultSet rests upon that, so arraysize ought to bear on performance.
>
> check-teamparticipation.py now slurps lots from the database, and one
> thing I tweaked to make it quicker is to hack the cursor arraysize to
> be much larger.
>
> I didn't take any measurements specifically about this because I was
> iterating other things too, but it appeared to have made slurping
> quicker. I'm kicking myself for not having taken proper isolated
> measurements, but they wouldn't be too hard to do now.
>
> Perhaps unbelievably, psycopg2 sets arraysize to 1. Storm attempts to
> salvage things by setting it to 10 (unless it has already been changed
> to something other than 1).

JamesH said previously that psycopg2 grabs everything from the DB
regardless, and that this is just cosmetic - this is when I was asking
for storm hooks to instrument the time it takes to get rows from the
resultset.

I suggest doing some measurements :)

-Rob