launchpad-dev team mailing list archive

[John Meinel <john@xxxxxxxxxxxxxxxxx>]

On Nov 16, 2011 8:35 PM, "Robert Collins" <robert.collins@xxxxxxxxxxxxx>
wrote:
>
> On Wed, Nov 16, 2011 at 11:59 PM, John Arbash Meinel
> <john@xxxxxxxxxxxxxxxxx> wrote:
> > I'm curious if you benchmarked bson vs json vs whatever the current
> > serializer is. I know I've heard a lot of statements that "bson is
> > faster", but it came up on the U1 mailing list and so I went ahead and
> > benchmarked it.
> >
> > json was encoded using simplejson.loads/dumps. It also supports having
> > a list as the top level object, so I made a string of 50k objects and
> > wrapped "[" + ",\n".join(lines) + "]".
> > I also compared it to decoding each entry one-by-one.
> >
> > My summary was:
> >
> >        encode_all      encode_by_1     decode_all      decode_by_1
> > json    0.766           1.130           0.511           0.795
> > bson                    0.873                           0.515
> >
> > Basically, at best bson is 1.5:1 faster than simplejson, but doing the
> > operation in bulk for simplejson was even faster than bjson. For both
> > encoding and decoding.
>
> Which bson encoder / decoder did you use? The mongodb one or the one
> on pypi? [idly curious]

Mongodb one. Pypi is pure python and 5-10x slower. Same for simplejson and
stdlib's json package.

>
> > This was 50k music metadata records, about 33MB of data in total.
> >
> > Also, I tried just writing the bson data out to a file, though I'm not
> > sure how you are supposed to write a list of bson records (maybe wrap
> > it in an object and put it as a value in a list?)
>
> bson requires a 'document' top level: {'mylist': [....]}.
>
> > I happen to really like than BSON is type-length-prefixed, but I
> > wonder if JSON wouldn't have had better interoperability.
>
> json doesn't handle non-unicode, which is an issue as we get some
> -real- crap in on HTTP, and also its very nice that bson supports
> datetime directly - we record a lot of datetimes ;)
>
> -Rob

Is JSON the one that has base60 floats? Or is that yaml?
John
=:->