software-store-developers team mailing list archive

[Timothy Arceri <t_arceri@xxxxxxxxxxxx>]

Hi Michael,
                I'm happy to hear you think this would make a good inclusion to 13.04. I have been using the ExecutionTime helper code a lot in tracking down bottle necks it is extremely useful. I have been tracking the bottlenecks I've found in this bug:  https://bugs.launchpad.net/ubuntu/+source/software-center/+bug/845579

Last night I tracked down some more bottlenecks in calls to Xapian. Specifically calls to get_mset() these calls take around 4 seconds in total on my machine.
Most of these calls are made from _blocking_perform_search() in softwarecenter/db/enquire.py
Aside from upstream optimizations to Xapian there doesn't 
appear to be any easy way around this. The maxitems parameter can be set to 0 to speedup get_mset() this results in only statistical information being returned which seems to be all that is use from these calls however the documentation says this also reduces the accuracy of the 
mset.get_matches_estimated() call which it seems the software center relies on being accurate so I dont know if change could be made.
So with the pickle load time and the Xapian calls that I have tracked so far that adds up to around 20-25% of the startup time.

Tim



----- Original Message -----
From: Michael Vogt <mvo@xxxxxxxxxx>
To: Timothy Arceri <t_arceri@xxxxxxxxxxxx>
Cc: "software-store-developers@xxxxxxxxxxxxxxxxxxx" <software-store-developers@xxxxxxxxxxxxxxxxxxx>
Sent: Wednesday, 10 October 2012 11:44 PM
Subject: Re: [Software-store-developers] Speeding up software center load times

On Sun, Oct 07, 2012 at 04:51:11PM -0700, Timothy Arceri wrote:
> Hi Guys,
Hi Timothy,

>             Due to time constraints I tend to limit my open source contributions to resolve personal annoyances. Lately I've turned my attention to the startup times on the Ubuntu Software Center which although is slightly better than when it was first release still takes far to long to load on my Netbook. 
> 
> 
> Anyway I'm tackling this one performance issue at a time and the first one that has come to my attention is that the software center is slow to load a Python pickle file that has been generated by apt-xapian-index. This file simply contains a serialized python dictionary and therefore its possible to use a more optimized serialization module than pickle. 
> 
> My suggestion is to replace pickle with msgpack which reduces load time (save times are also increased using msgpack ) on my machine from 1.4 seconds to 0.1 seconds also it reduces the filesize from 3.1MB to 1.7MB.

Woah, that sounds like a pretty impressive improvement. Thanks a bunch
for looking into this. 

> Obviously the drawback from my proposal is that it introduces a new dependency: msgpack-python 
> 

Unfortunately we are too late for the 12.10 release as the new
dependency needs to get a main approval first and get on the CD
etc. But as soon as 13.04 opens we can switch

> I have submitted a patch upstream to apt-xapian-index to make this change: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=689717

Looks like Enrico did not have the time yet to look at this, I am
happy to merge this myself, we may need a compatibilty pickle file for
some time and also convert the old data over. But that is probably
straightforward :)

> Anyway I would appreciate some feedback on the likely hood of adopting this improvement before I move on to tracking down further bottlenecks in software center.

I would certainly love to help with finding more bottlenecks. There is
a bit of helper code availalbe:
with ExecutionTime("description"):
     codeblock
for example and "software-center --debug-filter=performance" should
give some useful hints on areas that are slow.



Cheers,
Michael