zim-wiki team mailing list archive

[<HawKing@xxxxxxxxxxxxx>]

Would you mind pointing me to the source file(s) that manage this
indexing? I'd like to see if there is any way to speed the process
up for large numbers of files. 



On Mon, 3 Jul 2017 17:42:07 +0000
<HawKing@xxxxxxxxxxxxx> wrote:

> Yes, for medium sized notebooks, and those with a "normal" amount of
> files, indexing is still under 5 minutes. I also use Zim to manage a
> notebook under which there are lots of small work data files
> (>350,000).     
> 
> The progress bar suggests there is some part of the parsing process
> that slows down over time, as does a cursory check on the contents of
> the database updating over time. There are many more files added
> within the first few minutes, and many fewer over time, such that
> after a while, only one or two files are added ever several minutes.
> It suggests to me that the whole list is being re-processed or
> re-opened as part of the indexing loop, perhaps re-opening the sqlite
> file for every new file or something. Ultimately, I don't think that
> exponential slowdown is a necessity, but I have not had a free moment
> to familiarize myself with the source yet. 
> 
> Thanks!
> 
> 
> 
> On Mon, 03 Jul 2017 08:04:36 +0000
> Jaap Karssenberg <jaap.karssenberg@xxxxxxxxx> wrote:
> 
> > Yes, zim does indeed now build a tabel of all files in the notebook
> > folder, not just text files. However it doesn't access them, it just
> > stores file names and mtime.
> > 
> > Despite this change, the indexing is faster than with 0.65 in most
> > of my test cases. The behavior you describe suggest a huge amount of
> > files under the notebook folder, is this the case?
> > 
> > -- Jaap
> > 
> > On Sun, Jul 2, 2017 at 8:43 PM <HawKing@xxxxxxxxxxxxx> wrote:
> >   
> > > The notebooks that used to take me about 5 minutes to re-index are
> > > taking close to 40 hours for me (they are larger notebooks).
> > >
> > > It looks like the sql database is indexing every file under the
> > > root directory of the notebook, even those not associated with Zim
> > > directly, like zip or data files. I'm not sure if that was
> > > happening with earlier versions.
> > >
> > >
> > >
> > > On Sat, 1 Jul 2017 23:32:16 +0200
> > > Olivier Boesch <boesch@xxxxxxx> wrote:
> > >    
> > > > 6 minutes to reindex. pretty long in comparison with the 0.65.
> > > >
> > > >
> > > > Le 01/07/2017 à 23:24, Olivier Boesch a écrit :    
> > > > >
> > > > > I seem to experience the same issue...
> > > > >
> > > > > I clicked the "cancel" button after several minutes...
> > > > >
> > > > > testing now how long it takes to re-index...
> > > > >
> > > > >
> > > > > Le 01/07/2017 à 23:04, HawKing@xxxxxxxxxxxxx a écrit :    
> > > > >> After this latest upgrade came through (it looks great),
> > > > >> notebooks that took me several minutes to re-index are now
> > > > >> taking multiple days of time, and it seems like an
> > > > >> exponential slowdown with the number (and maybe size) of
> > > > >> files under the notebook root directory. Has anyone else
> > > > >> experienced this?
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> Mailing list:https://launchpad.net/~zim-wiki
> > > > >> Post to     :zim-wiki@xxxxxxxxxxxxxxxxxxx
> > > > >> Unsubscribe :https://launchpad.net/~zim-wiki
> > > > >> More help   :https://help.launchpad.net/ListHelp    
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Mailing list: https://launchpad.net/~zim-wiki
> > > > > Post to     : zim-wiki@xxxxxxxxxxxxxxxxxxx
> > > > > Unsubscribe : https://launchpad.net/~zim-wiki
> > > > > More help   : https://help.launchpad.net/ListHelp    
> > > >    
> > >
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~zim-wiki
> > > Post to     : zim-wiki@xxxxxxxxxxxxxxxxxxx
> > > Unsubscribe : https://launchpad.net/~zim-wiki
> > > More help   : https://help.launchpad.net/ListHelp
> > >    
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~zim-wiki
> Post to     : zim-wiki@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~zim-wiki
> More help   : https://help.launchpad.net/ListHelp