← Back to team overview

zim-wiki team mailing list archive

Re: zim usage of linux threads

 

Quick update about the subject topic:

I applied the suggested hack immediately on November 20, and got the associated "No file monitor support - changes will go undetected" at zim start.

After some 25 days running with no file monitoring zim is reported to use 261 threads. While the number grew much slower than before, it still looks large to me.

Maybe there is something wrong in the underlying python services? Being totally phyton illiterate, I wonder if it is possible to to verify this with a simple python test program.

Anyone willing to help?

Thanks,
mario

On 11/20/2014 07:08 AM, Jaap Karssenberg wrote:
You can hack zim to stop listening for filesystem signals. If that makes the problem go away, you can at least confirm the root cause.

To do so edit "zim/fs.py" -- this will be e.g. in /usr/lib/python/zim - depends on your install locations.

In this file near the top there is a section:

-------------
try:
    import gobject
    import gio
    if not gio.File.trash:
        gio = None
except ImportError:
    pass
--------------

If you remove this section, zim stops using file system monitors.

Hope this helps,

Jaap




On Mon, Nov 17, 2014 at 4:34 PM, mario@tiscali <mbezzi@xxxxxxxxxx <mailto:mbezzi@xxxxxxxxxx>> wrote:

    Jaap thank you for taking the time to look into this.

    I also have the feeling that this has to do with filesystem's
    change notifications.

    While zim became a very important part of my everyday's actvity,
    there are days worse than others, and today is one of these: I
    have been editing files in zim's tree (not necessarily zim's
    pages, rather files attached to zim's pages) quite a lot (and no
    FTP!), and here we go:

    bash: fork: retry: Resource temporarily unavailable
    bash: fork: retry: Resource temporarily unavailable
    bash: fork: retry: Resource temporarily unavailable

    I am continuosly running the following command to keep track of
    what's going on:

    $ while true; do exec ps -u mario -To comm | sort | uniq -c | sort
    -n| tail;ps -u mario_bezzi -To
    pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm|wc
    -l;sleep 10; done

    and this is the result (showing only the major contributors:

          5 nautilus
          5 totem
          7 gnome-keyring-d
          8 spicec
         12 taskldr
         23 dropbox
        ... ...
        629 zim
       1017 Total

    Which means that at the moment zim is using 629 threads driving
    the total very close to the limit (1024), and from time to time to
    the limit, which causes the fork failure.

    This is RHEL 6.5 with current maintenance. I would like to help
    gathering diagnostics if I only knew how.

    Hints?

    Thanks again,
    mario


    On 11/05/2014 08:13 AM, Jaap Karssenberg wrote:
    Hi Mario,

    Just checked it on my own system. Just running one notebook in
    stand-alone mode I get only 2 threads, the main thread + one for
    the thumbnails in the attachment browser. When running with
    background daemon I get ~5 threads (they show up as "python" in
    the list, not "zim" so not sure exactly which belong to zim and
    which don't). I think this is because we use the
    "multiprocessing" library, which uses a few threads per process
    for process communication.

    So nothing weird there, although more extensive testing would
    probably be needed to be sure I can not duplicate your result.

    As to what may be causing it, somehow the ftp process must be
    pinging the zim process. Maybe the filesystem starts notifying
    zim of file changes?  Should normally not happen if the ftp
    transfer goes to a different folder, but not sure what could be
    happenign on that layer. Just a wild theory.

    Regards,

    Jaap


    On Sat, Oct 25, 2014 at 11:36 AM, mario@tiscali
    <mbezzi@xxxxxxxxxx <mailto:mbezzi@xxxxxxxxxx>> wrote:

        Hi Jaap,

            just a quick update: After recycling zim-wiki the number
        of threads stayed in the 11 ballpark for quite a long time.
        It suddenly started climbing while I was running a completely
        unrelated ftp file transfer. Stopped climbing when the ftp
        ended, restarted climbing when I purposedly ran another ftp.
        It is now at 227!

        Can't even imagine the relationship between zim-wiki and my
        ftp activity, but I wanted to share what I saw so far.

        Thank you,
        mario

        On 10/24/2014 07:09 PM, mario@tiscali wrote:
        Jaap,

            thanks for your kind reply.

        To just get the number of active threads issue the following
        command: ps -eT|grep zim|wc -l

        For a detailed list use: ps -fT -C zim

        By the way: After a quit/start and couple of page views
        zim-wiki sits at 11 threads. I will keep it under control.

        Long and probably not nicely formatted command output follows:

        ..snip..






References