launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

> 
> The profile showed 16 seconds (of 25 up to the kill-point) going into
> storm object deserialisation. 30000 objects. (Thats 3000 rows, 10
> objects per row).
> 

One possibility comes to mind here. Deserialisation tends to be the time
when you are creating a lot of python objects, correct? Creating lots of
objects without getting rid of them triggers GC runs. GC runs tend to
depend on the total size of all objects in the VM, not restricted to
just your current thread, etc. (In theory, the generational GC only
depends on 'new enough' objects, but once you trigger enough level 1
runs, you do get a level 2 run, etc.)

30k objects isn't all that many, but depending on the objects, they may
be multi-layered. (A plain Python instance-of-a-class is the object +
its dict). This was only really noticeable in bzr when we had >1M tuple
objects around, and I don't know the internals of launchpad servers.

Note also that python2.7 is supposed to help with the quadratic tuple +
dict behavior. Once GC runs, if it sees that a given tuple only points
at non-gc objects (strings, or other 'fixed' tuples) then it removes the
tuple (or dict) from GC.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyGX8wACgkQJdeBCYSNAAO0FgCgpLgE5UcKIf2QYkhsjcJL8k5l
/tQAoJ/xfk0Xsoin8Zeoe90BdHI6YTTT
=3EdE
-----END PGP SIGNATURE-----