oship-dev team mailing list archive

[Tim Cook <timothywayne.cook@xxxxxxxxx>]

Hi All,

I have forwarded the email below so that some of you that haven't used
the ZODB before can get an idea of it's capabilities.

A lot of people ask me ability speed and scalability.

Regards,
Tim

-------- Original Message --------
Subject: Re: [ZODB-Dev] Some interesting (to some:) numbers
Date: Tue, 11 May 2010 14:36:46 -0400
From: Jim Fulton <jim@xxxxxxxx>
To: Roel Bruggink <roel@xxxxxxxxxxxxx>
CC: zodb-dev <zodb-dev@xxxxxxxx>

On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink <roel@xxxxxxxxxxxxx> wrote:
> On Sun, May 9, 2010 at 8:33 PM, Jim Fulton <jim@xxxxxxxx> wrote:
>>
>> Our recent discussion of compression made me curious so I did some
>> analysis of pickle sizes in one of our large databases. This is for a
>> content management system.  The database is packed weekly.  It doesn't
>> include media, which are in blobs.
>>
>> There were ~19 million transaction in the database and around 130
>> million data records. About 60% of the size was taken up by BTrees.
>> Compressing pickles using zlib with default compression reduced the
>> pickle sizes by ~58%. The average uncompressed record size was 1163
>> bytes.  The average compressed size was ~493 bytes.
>>
>> This is probably enough of a savings to make compression interesting.

...

> That's really interesting! Did you notice any issues performance wise, or
> didn't you check that yet?

OK, I did some crude tests.  It looks like compressing is a little
less expensive than pickling and decompressing is a little more
expensive than unpickling, which is to say this is pretty cheap.  For
example, decompressing a data record took around 20 microseconds on my
machine. A typical ZEO load takes 10s of milliseconds. Even in Shane's
zodb shootout benchmark which loads data from ram, load times are
several hundred microseconds or more.

I don't think compression will hurt performance.  It is likeley to
help it in practice because:

- There will be less data to send back and forth to remote servers.

- Smaller databases will get more benefit from disk caches.
  (Databases will be more likely to fit on ssds.)

- ZEO caches (and relstorage memcached caches) will be able to hold
  more object records.

Jim

--
Jim Fulton
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@xxxxxxxx
https://mail.zope.org/mailman/listinfo/zodb-dev


-- 
***************************************************************
Timothy Cook, MSc

LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
Skype ID == (upon request)
Academic.Edu Profile: http://uff.academia.edu/TimothyCook

You may get my Public GPG key from  popular keyservers or
from this link http://timothywayne.cook.googlepages.com/home