gtg-contributors team mailing list archive
-
gtg-contributors team
-
Mailing list archive
-
Message #00038
Re: Refactorisation progress : It works (but it's slow)
On Thu, Feb 25, 2010 at 9:25 AM, Lionel Dricot <ploum@xxxxxxxxx> wrote:
>
> Thanks for your input. I didn't touch anything related to tags but,
> indeed, it seems that it is sometimes broken. After some test, it appears
> that the task view are, in fact, quite efficient. I did some very very
> basic optimisations yesterday and I'm currently between 100% and 200% of
> the trunk loading time (when I wrote the mail yesterday, we were at 1000%
> so it looks like the concept is good).
>
> The problem is that taskbrowser was the root of everything. Everything was
> in it, from filtering tasks/tags to windows management and tasks updating.
>
> Every filtering feature should be moved to the filterbank. I will create a
> new module, called view_manager, that will contains the current manager.py
> in core and will be responsible of loading the preferences and launching
> the gtk loop.
>
> Also, it looks like the tag store itself might benefit from some
> refactoring and some love.
>
>
> Bertrand : I removed the refresh because it's not needed anymore. That's
> the beauty of all of that. The only global refreshes now happen when a
> filter is applied/unapplied. We will even be able to reduce the artificial
> 7 second delay before updating that we introduced in the TaskEditor.
>
>
> PS : I'm quite bad at undestanding profiling output, so thanks for your
> interpretation. I remember that you found a visualisation tool. What was
> the name again ?
It's runsnakerun. But here's the deal:
- cumulative time is the total time spent in a function, including
the time spent in its callees. Accordingly, we spent the most time in
gtg.main and such since they're the ones that launch everything
- total time is the total time spent ONLY in the function itself, not
in its callees.
- percall, is the mean time per call
So you generally must have a look at cumulative time, and try to
identify what the worst contributor is. For instance, on_task_modified
time is heavily due to update_tags_for_task, which is called inside
it.
Having a look at the most called functions is also worth it, since you
really want ot optimize functions that are called a lot of time (e.g.
get_value() )
>
> On Thu, 25 Feb 2010 01:27:16 +0100, Bertrand Rousseau
> <bertrand.rousseau@xxxxxxxxx> wrote:
>> On Wed, Feb 24, 2010 at 5:57 PM, Lionel Dricot <ploum@xxxxxxxxx> wrote:
>>>
>>> I've very good news for myself : I see the light.
>>>
>>> The FilteredTree is now working nearly perfectly. It is rock-solid,
> even
>>> when loading 300 tasks at the same time. I also implemented the Closed
>>> Tasks pane with it. The FiltersBank is also working great.
>>
>> Great news!
>>
>>> Bad news : with a lot of tasks, the start time is really really really
>>> slow. (like 10 times the current trunk launch time). On the other hand,
>>> it
>>> looks like other performance issues are mostly solved (like renaming a
>>> subtask in a task).
>>
>> Task renaming is maybe due to the fact that you disabled the refresh
>> methods from the browser. Originally, it can from the fact that
>> browser refresh is called at each letter you type by the editor. It
>> should be worth looking at this too or it will probably show again
>> sometime.
>>
>>> So now, I need *you*. I need testing and profiling.
>>>
>>> Do : bzr branch lp:~gtg/gtg/gtg-refactor
>>> and launch it first with debugging data then with actual data. Do some
>>> profiling ( http://live.gnome.org/gtg/development ) and send me the
>>> results.
>>
>> Here's mine (on my netbook, so slownesses are sensible here):
>>
>> Ordered by: cumulative time
>> List reduced from 947 to 15 due to restriction <15>
>>
>> ncalls tottime percall cumtime percall filename:lineno(function)
>> 1 0.000 0.000 10.889 10.889
>> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:91(main)
>> 1 0.000 0.000 10.127 10.127
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/manager.py:49(show_browser)
>> 1 0.000 0.000 9.047 9.047
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1795(main)
>> 1 4.523 4.523 8.653 8.653 {gtk._gtk.main}
>> 244 0.030 0.000 3.011 0.012
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1642(on_task_modified)
>> 244 0.027 0.000 2.595 0.011
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tagtree.py:53(update_tags_for_task)
>> 4032 0.263 0.000 2.457 0.001 {method 'row_changed' of
>> 'gtk.TreeModel' objects}
>> 7208 0.394 0.000 1.734 0.000 {method 'get_value' of
>> 'gtk.TreeModel' objects}
>> 7495 0.060 0.000 1.380 0.000
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tagtree.py:82(on_get_value)
>> 732 0.023 0.000 1.347 0.002
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:788(tag_visible_func)
>> 879 0.297 0.000 1.295 0.001
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/tagstore.py:305(get_tasks_nbr)
>> 12660 0.107 0.000 1.140 0.000
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/requester.py:135(get_task)
>> 1 0.000 0.000 1.074 1.074
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:83(__init__)
>> 12660 0.231 0.000 1.034 0.000
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/datastore.py:51(get_task)
>> 2709 0.175 0.000 0.825 0.000
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:858(tag_sort_func)
>>
>> As you can see, a lot of time is spent in update_tags_for_task. If you
>> comment this out, get_value and on_task_modified don't even show up in
>> the profiling log:
>>
>> Ordered by: cumulative time
>> List reduced from 944 to 15 due to restriction <15>
>>
>> ncalls tottime percall cumtime percall filename:lineno(function)
>> 1 0.000 0.000 9.428 9.428
>> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:91(main)
>> 1 0.000 0.000 8.639 8.639
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/manager.py:49(show_browser)
>> 1 0.000 0.000 7.443 7.443
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1795(main)
>> 1 5.892 5.892 7.012 7.012 {gtk._gtk.main}
>> 1 0.001 0.001 1.173 1.173
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:83(__init__)
>> 1 0.000 0.000 0.776 0.776
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:525(_init_plugin_engine)
>> 1 0.002 0.002 0.775 0.775
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:147(load_plugins)
>> 12 0.002 0.000 0.587 0.049
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:40(__init__)
>> 12 0.001 0.000 0.544 0.045
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:108(_load_module)
>> 12 0.003 0.000 0.514 0.043 {imp.load_module}
>> 29 0.003 0.000 0.442 0.015
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/tools/cleanxml.py:98(savexml)
>> 2 0.000 0.000 0.430 0.215
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/core/__init__.py:160(save_datastore)
>> 10 0.356 0.036 0.395 0.040 {method 'show' of
>> 'gtk.Widget' objects}
>> 975 0.064 0.000 0.383 0.000
>>
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tasktree.py:94(on_get_value)
>> 29 0.001 0.000 0.334 0.012
>> /usr/lib/python2.6/xml/dom/minidom.py:47(toprettyxml)
>>
>> It's probably not the task of the browser to refresh each tag related
>> to a task. Since Tags have references to their tasks, they are
>> modified as well when a Task is edited. So we should probably let the
>> tag emit a signal to warn everyone they changed (much like the Task
>> are doing now). Let the objets which care about it to register
>> themselve to this signal through the requester.
>>
>> That would imply having a "on_tag_modified" in the browser that would
>> trigger some job to perform on the browser (probably not much since,
>> if I understood well, most of the filtering work is now done outside
>> the browser)
>>
>>> If you want to contribute, there's a huge cleaning to do in the
>>> browser.py
>>> ! A lot of cruft should be removed (and I don't understand everything
>>> there).
>>
>> I agree with what you say, but it would be more efficient if you could
>> be more specific and point out some of these cruft.
>>
>>>
>>> What does not work :
>>> - tag filtering. They are implemented. We just need to do
>>> "self.req.apply_filter(tagname)" when a particular tag is selected but
> I
>>> don't know where it is in browser.py and I'm a bit tired. (any help is
>>> welcome here)
>>> - drag-n-drop : code is commented out in tasktree.py. Maybe it will
> work
>>> without any trouble.
>>> - multiparent (not tested yet)
>>> - tag count in tagtree
>>>
>>> What is broken :
>>> - plugins and plugin_api
>>> - dbus interface
>>>
>>>
>>> I was expecting better performances but FilteredTree appeared to be a
> lot
>>> more complex than expected. So now, it's time to optimize it a bit and
>>> I'm
>>> wondering where is the bottleneck.
>>
>> Like I wrote above, it seems to be situated around tags update
>> mechanisms and TagTreeModel updates.
>>
>>>
>>> Thanks for listenning,
>>>
>>>
>>> Lionel
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~gtg-contributors
>>> Post to : gtg-contributors@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~gtg-contributors
>>> More help : https://help.launchpad.net/ListHelp
>>>
>
--
Bertrand Rousseau
References