zim-wiki team mailing list archive

[<HawKing@xxxxxxxxxxxxx>]

Yes, for medium sized notebooks, and those with a "normal" amount of
files, indexing is still under 5 minutes. I also use Zim to manage a
notebook under which there are lots of small work data files
(>350,000).   

The progress bar suggests there is some part of the parsing process that
slows down over time, as does a cursory check on the contents of the
database updating over time. There are many more files added within the
first few minutes, and many fewer over time, such that after a while,
only one or two files are added ever several minutes. It suggests to me
that the whole list is being re-processed or re-opened as part of the
indexing loop, perhaps re-opening the sqlite file for every new file or
something. Ultimately, I don't think that exponential slowdown is a
necessity, but I have not had a free moment to familiarize myself with
the source yet. 

Thanks!



On Mon, 03 Jul 2017 08:04:36 +0000
Jaap Karssenberg <jaap.karssenberg@xxxxxxxxx> wrote:

> Yes, zim does indeed now build a tabel of all files in the notebook
> folder, not just text files. However it doesn't access them, it just
> stores file names and mtime.
> 
> Despite this change, the indexing is faster than with 0.65 in most of
> my test cases. The behavior you describe suggest a huge amount of
> files under the notebook folder, is this the case?
> 
> -- Jaap
> 
> On Sun, Jul 2, 2017 at 8:43 PM <HawKing@xxxxxxxxxxxxx> wrote:
> 
> > The notebooks that used to take me about 5 minutes to re-index are
> > taking close to 40 hours for me (they are larger notebooks).
> >
> > It looks like the sql database is indexing every file under the root
> > directory of the notebook, even those not associated with Zim
> > directly, like zip or data files. I'm not sure if that was happening
> > with earlier versions.
> >
> >
> >
> > On Sat, 1 Jul 2017 23:32:16 +0200
> > Olivier Boesch <boesch@xxxxxxx> wrote:
> >  
> > > 6 minutes to reindex. pretty long in comparison with the 0.65.
> > >
> > >
> > > Le 01/07/2017 à 23:24, Olivier Boesch a écrit :  
> > > >
> > > > I seem to experience the same issue...
> > > >
> > > > I clicked the "cancel" button after several minutes...
> > > >
> > > > testing now how long it takes to re-index...
> > > >
> > > >
> > > > Le 01/07/2017 à 23:04, HawKing@xxxxxxxxxxxxx a écrit :  
> > > >> After this latest upgrade came through (it looks great),
> > > >> notebooks that took me several minutes to re-index are now
> > > >> taking multiple days of time, and it seems like an exponential
> > > >> slowdown with the number (and maybe size) of files under the
> > > >> notebook root directory. Has anyone else experienced this?
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> Mailing list:https://launchpad.net/~zim-wiki
> > > >> Post to     :zim-wiki@xxxxxxxxxxxxxxxxxxx
> > > >> Unsubscribe :https://launchpad.net/~zim-wiki
> > > >> More help   :https://help.launchpad.net/ListHelp  
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Mailing list: https://launchpad.net/~zim-wiki
> > > > Post to     : zim-wiki@xxxxxxxxxxxxxxxxxxx
> > > > Unsubscribe : https://launchpad.net/~zim-wiki
> > > > More help   : https://help.launchpad.net/ListHelp  
> > >  
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~zim-wiki
> > Post to     : zim-wiki@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~zim-wiki
> > More help   : https://help.launchpad.net/ListHelp
> >