gtg-contributors team mailing list archive

[Lionel Dricot <ploum@xxxxxxxxx>]

Normal. 

When you change the parent relationship of a tasks, this task is updated,
sending the updated signal and saving itself in the backend. The modified
signal is also sent to the children and the parents of that particular
task. All those modified-signal are catched by che tagstore to update
itself.

The problem was that the "add_parent" method was not checking the previous
situation. Thus, tasks were doing all that work even if the add_parent was
adding a parent that previously existed and thus were not changing
anything. This is particularly evident during the load time because, when
reconstructing a tree, we use the "add_parent" method (even if the task
already knows about its parent).

What I simply did was to add a check to the add_parent method so it does
nothing if the parent was already there. Results ? On my 300 tasks, during
load, 100 were saved to the backend (yes, 100 file-write operation) and
3000 modified signals were sent.

After adding this check, we are at 0-0, which is perfect :-)

There are still some rooms for improvement : the tagstore, the fact that
the task.sync is called a bit too much (3 times when opening a task) but,
at least, we are in a shape which is 1000 times better than the trunk.
(once I've solved the tag issue).


On Thu, 25 Feb 2010 11:00:52 +0100, Bertrand Rousseau
<bertrand.rousseau@xxxxxxxxx> wrote:
> On Thu, Feb 25, 2010 at 10:14 AM, Lionel Dricot <ploum@xxxxxxxxx> wrote:
>>
>> Only for performances, could you please take the latest revision (658)
of
>> my refactoring branch and try it with a lot of tasks and confirm me
that
>> I'm not dreaming ?
> 
> It seems quite fast, yes, but it's not using update_tags_for_task
either:
> 
>          109098 function calls (105962 primitive calls) in 3.730 CPU
>          seconds
> 
>    Ordered by: cumulative time
>    List reduced from 1042 to 15 due to restriction <15>
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>         1    0.000    0.000    3.730    3.730
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:91(main)
>         1    0.000    0.000    3.567    3.567
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/manager.py:49(show_browser)
>         1    0.000    0.000    3.428    3.428
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1795(main)
>         1    3.313    3.313    3.331    3.331 {gtk._gtk.main}
>         1    0.000    0.000    0.139    0.139
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:83(__init__)
>        10    0.093    0.009    0.096    0.010 {method 'show' of
> 'gtk.Widget' objects}
>         1    0.000    0.000    0.069    0.069
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:525(_init_plugin_engine)
>         1    0.000    0.000    0.068    0.068
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:147(load_plugins)
>        46    0.001    0.000    0.068    0.001
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/tools/cleanxml.py:98(savexml)
>        12    0.000    0.000    0.065    0.005
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:40(__init__)
>        12    0.000    0.000    0.062    0.005
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:108(_load_module)
>         2    0.000    0.000    0.061    0.031
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/__init__.py:160(save_datastore)
>        12    0.001    0.000    0.060    0.005 {imp.load_module}
>         1    0.000    0.000    0.049    0.049
> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:66(check_instance)
>         2    0.000    0.000    0.045    0.023
>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/datastore.py:203(quit)
> 
>> I just can't believe what I'm seeing but that computer is fast.
>>
>>
>> Lionel
>>
>> PS: tags display is still broken.
>>
>>
>>
>> On Thu, 25 Feb 2010 01:27:16 +0100, Bertrand Rousseau
>> <bertrand.rousseau@xxxxxxxxx> wrote:
>>> On Wed, Feb 24, 2010 at 5:57 PM, Lionel Dricot <ploum@xxxxxxxxx>
wrote:
>>>>
>>>> I've very good news for myself : I see the light.
>>>>
>>>> The FilteredTree is now working nearly perfectly. It is rock-solid,
>> even
>>>> when loading 300 tasks at the same time. I also implemented the
Closed
>>>> Tasks pane with it. The FiltersBank is also working great.
>>>
>>> Great news!
>>>
>>>> Bad news : with a lot of tasks, the start time is really really
really
>>>> slow. (like 10 times the current trunk launch time). On the other
hand,
>>>> it
>>>> looks like other performance issues are mostly solved (like renaming
a
>>>> subtask in a task).
>>>
>>> Task renaming is maybe due to the fact that you disabled the refresh
>>> methods from the browser. Originally, it can from the fact that
>>> browser refresh is called at each letter you type by the editor. It
>>> should be worth looking at this too or it will probably show again
>>> sometime.
>>>
>>>> So now, I need *you*. I need testing and profiling.
>>>>
>>>> Do : bzr branch lp:~gtg/gtg/gtg-refactor
>>>> and launch it first with debugging data then with actual data. Do
some
>>>> profiling ( http://live.gnome.org/gtg/development ) and send me the
>>>> results.
>>>
>>> Here's mine (on my netbook, so slownesses are sensible here):
>>>
>>>    Ordered by: cumulative time
>>>    List reduced from 947 to 15 due to restriction <15>
>>>
>>>    ncalls  tottime  percall  cumtime  percall
filename:lineno(function)
>>>         1    0.000    0.000   10.889   10.889
>>> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:91(main)
>>>         1    0.000    0.000   10.127   10.127
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/manager.py:49(show_browser)
>>>         1    0.000    0.000    9.047    9.047
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1795(main)
>>>         1    4.523    4.523    8.653    8.653 {gtk._gtk.main}
>>>       244    0.030    0.000    3.011    0.012
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1642(on_task_modified)
>>>       244    0.027    0.000    2.595    0.011
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tagtree.py:53(update_tags_for_task)
>>>      4032    0.263    0.000    2.457    0.001 {method 'row_changed' of
>>> 'gtk.TreeModel' objects}
>>>      7208    0.394    0.000    1.734    0.000 {method 'get_value' of
>>> 'gtk.TreeModel' objects}
>>>      7495    0.060    0.000    1.380    0.000
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tagtree.py:82(on_get_value)
>>>       732    0.023    0.000    1.347    0.002
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:788(tag_visible_func)
>>>       879    0.297    0.000    1.295    0.001
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/tagstore.py:305(get_tasks_nbr)
>>>     12660    0.107    0.000    1.140    0.000
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/requester.py:135(get_task)
>>>         1    0.000    0.000    1.074    1.074
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:83(__init__)
>>>     12660    0.231    0.000    1.034    0.000
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/datastore.py:51(get_task)
>>>      2709    0.175    0.000    0.825    0.000
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:858(tag_sort_func)
>>>
>>> As you can see, a lot of time is spent in update_tags_for_task. If you
>>> comment this out, get_value and on_task_modified don't even show up in
>>> the profiling log:
>>>
>>>    Ordered by: cumulative time
>>>    List reduced from 944 to 15 due to restriction <15>
>>>
>>>    ncalls  tottime  percall  cumtime  percall
filename:lineno(function)
>>>         1    0.000    0.000    9.428    9.428
>>> /home/rousseau/workspace/gtg/gtg-refactor/GTG/gtg.py:91(main)
>>>         1    0.000    0.000    8.639    8.639
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/manager.py:49(show_browser)
>>>         1    0.000    0.000    7.443    7.443
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:1795(main)
>>>         1    5.892    5.892    7.012    7.012 {gtk._gtk.main}
>>>         1    0.001    0.001    1.173    1.173
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:83(__init__)
>>>         1    0.000    0.000    0.776    0.776
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/browser.py:525(_init_plugin_engine)
>>>         1    0.002    0.002    0.775    0.775
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:147(load_plugins)
>>>        12    0.002    0.000    0.587    0.049
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:40(__init__)
>>>        12    0.001    0.000    0.544    0.045
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/plugins/engine.py:108(_load_module)
>>>        12    0.003    0.000    0.514    0.043 {imp.load_module}
>>>        29    0.003    0.000    0.442    0.015
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/tools/cleanxml.py:98(savexml)
>>>         2    0.000    0.000    0.430    0.215
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/core/__init__.py:160(save_datastore)
>>>        10    0.356    0.036    0.395    0.040 {method 'show' of
>>> 'gtk.Widget' objects}
>>>       975    0.064    0.000    0.383    0.000
>>>
>>
/home/rousseau/workspace/gtg/gtg-refactor/GTG/taskbrowser/tasktree.py:94(on_get_value)
>>>        29    0.001    0.000    0.334    0.012
>>> /usr/lib/python2.6/xml/dom/minidom.py:47(toprettyxml)
>>>
>>> It's probably not the task of the browser to refresh each tag related
>>> to a task. Since Tags have references to their tasks, they are
>>> modified as well when a Task is edited. So we should probably let the
>>> tag emit a signal to warn everyone they changed (much like the Task
>>> are doing now). Let the objets which care about it to register
>>> themselve to this signal through the requester.
>>>
>>> That would imply having a "on_tag_modified" in the browser that would
>>> trigger some job to perform on the browser (probably not much since,
>>> if I understood well, most of the filtering work is now done outside
>>> the browser)
>>>
>>>> If you want to contribute, there's a huge cleaning to do in the
>>>> browser.py
>>>> ! A lot of cruft should be removed (and I don't understand everything
>>>> there).
>>>
>>> I agree with what you say, but it would be more efficient if you could
>>> be more specific and point out some of these cruft.
>>>
>>>>
>>>> What does not work :
>>>> - tag filtering. They are implemented.  We just need to do
>>>> "self.req.apply_filter(tagname)" when a particular tag is selected
but
>> I
>>>> don't know where it is in browser.py and I'm a bit tired. (any help
is
>>>> welcome here)
>>>> - drag-n-drop : code is commented out in tasktree.py. Maybe it will
>> work
>>>> without any trouble.
>>>> - multiparent (not tested yet)
>>>> - tag count in tagtree
>>>>
>>>> What is broken :
>>>> - plugins and plugin_api
>>>> - dbus interface
>>>>
>>>>
>>>> I was expecting better performances but FilteredTree appeared to be a
>> lot
>>>> more complex than expected. So now, it's time to optimize it a bit
and
>>>> I'm
>>>> wondering where is the bottleneck.
>>>
>>> Like I wrote above, it seems to be situated around tags update
>>> mechanisms and TagTreeModel updates.
>>>
>>>>
>>>> Thanks for listenning,
>>>>
>>>>
>>>> Lionel
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~gtg-contributors
>>>> Post to     : gtg-contributors@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~gtg-contributors
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~gtg-contributors
>> Post to     : gtg-contributors@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~gtg-contributors
>> More help   : https://help.launchpad.net/ListHelp
>>