calibre-devs team mailing list archive

[Kovid Goyal <kovid@xxxxxxxxxxxxxx>]

On Sun, Nov 21, 2010 at 10:13:41PM -0600, Chad Crawley wrote:
> My catalog is about 30,000 records, and when I start Calibre, I spend about
> five minutes looking at the splash screen. So I ran it through cProfile, and
> a couple of things stand out.
> 
> The biggest item is over three minutes on DatabaseLibrary2.has_cover(). I
> tried replacing os.access() with os.path.exists(), but it only shaves off 15
> seconds. Now I'm wondering, exactly what are the ramifications of removing
> the call from DatabaseLibrary2.get_data_as_dict()? It looks like the actual
> cover display does its own check anyway.

has_cover is used for cover:false searches. The proper fix is to modify the books table adding a has_cover column, which is then updated by the set_cover and remove_cover methods.

> 
> The second item looks like SQLite performance related to the sortconcat()
> aggregate. Is the sortconcat() the only reason meta2 is not a persistent
> view? When I create a persistent view using a derived table* instead of
> sortconcat(), it shaves off about 30 seconds of load time.
> 
> 
> *(SELECT group_concat(name) FROM (
>     SELECT name FROM books_authors_link AS link INNER JOIN
>     authors ON(link.author=authors.id) WHERE link.book=books.id
>     ORDER BY name ASC)
>     names)
> authors
> 

No, it's temporary for various architectural reasons. That is not going to change. You can speed that up by re-implementing sort_concat in C. But there may be some issues with registering the function with sqlite in a way that makes it available to the python sqlite module as well. I'm not familiar enough with the sqlite C API to comment on that. 

Kovid.

-- 
_____________________________________

Dr. Kovid Goyal 
http://www.kovidgoyal.net
http://calibre-ebook.com
_____________________________________

Attachment: pgp5TbUc7syzQ.pgp
Description: PGP signature