launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

> Technically, the bson change is a change to the default serializer
> in oops_datedir_repo, so if you bump the rev of that component to
> 0.0.8 or above and *don't* want to switch to bson, be sure to
> update your config to pass in the rfc822 serializer.
...

I'm curious if you benchmarked bson vs json vs whatever the current
serializer is. I know I've heard a lot of statements that "bson is
faster", but it came up on the U1 mailing list and so I went ahead and
benchmarked it.

json was encoded using simplejson.loads/dumps. It also supports having
a list as the top level object, so I made a string of 50k objects and
wrapped "[" + ",\n".join(lines) + "]".

I also compared it to decoding each entry one-by-one.

My summary was:

	encode_all	encode_by_1	decode_all	decode_by_1
json	0.766		1.130		0.511		0.795
bson			0.873				0.515

Basically, at best bson is 1.5:1 faster than simplejson, but doing the
operation in bulk for simplejson was even faster than bjson. For both
encoding and decoding.

This was 50k music metadata records, about 33MB of data in total.

Also, I tried just writing the bson data out to a file, though I'm not
sure how you are supposed to write a list of bson records (maybe wrap
it in an object and put it as a value in a list?)

Anyway:

$ \ls -s music_metadata.*
33744 music_metadata.bson  33464 music_metadata.json

The JSON files were actually smaller than the BSON in the end. My
guess is that length prefixing everything takes more bytes than the
{,},",: characters.

I happen to really like than BSON is type-length-prefixed, but I
wonder if JSON wouldn't have had better interoperability.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7Dl6EACgkQJdeBCYSNAAOVtACfV7U8tL9O1Y4hqk3/Sn1Lv/pd
SwoAmwba6f0HDfCXOCymWbzn8iv0kgpu
=VTk8
-----END PGP SIGNATURE-----