syncany-team team mailing list archive

[Philipp Heckel <philipp.heckel@xxxxxxxxx>]

Hi Jeroen,


I read that one of the goals was removing JPA from the project, because
> it's too slow.
> So I guess that
>
>    1. you want to write queries yourself without JPA
>    2. *OR*
>    3. you also want to get rid of the database, which is I believe Apache
>    Derby?
>
> What's the alternative? Saving the indexes into plain text files?
>

The problem is not _saving_ the index, it's the thousands of queries
Syncany has to make during the chunking process: for each chunk, the
chunking algorithm has to ask the index "is this chunk there yet (SELECT)?
If not, add it (INSERT)". Assuming a 400 MB file and 4 KB chunks, that
would be 100.000 queries (+ 100.000 INSERTs if the file is new). With an
SQL based index, this is slow. In fact, I assume that any SQL based DB is
slower than a simple hash table in the memory. And as long as the chunk
index doesn't grow too big, we're good with that approach.

Take this example: you store 10 GB of data in the Syncany folder. With a
chunk size of 4 KB, that's 10 x 1000 x1000 KB / 4 KB = 2.5 million chunks.
Assuming a memory usage of 50 byte (checksum of the chunk + references),
that's 2.5 million x 50 bytes = 125 MB -- which is not low, but okay
compared to some other applications (*cough* Firefox *cough*).

Since we need to exchange the index with the other clients anyway, we do
need to store it in a custom file format.

Good for an answer? :-)

Cheers,
Philipp

>
> Sincerely,
>
> Jeroen Verheye
>
> --
> Mailing list: https://launchpad.net/~syncany-team
> Post to     : syncany-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~syncany-team
> More help   : https://help.launchpad.net/ListHelp
>
>