launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Wed, Oct 5, 2011 at 2:36 AM, Aaron Bentley <aaron@xxxxxxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11-10-04 05:54 AM, Robert Collins wrote:
>> Actually thats a lie, this mail is about changes that have already
>> happened, *and* about some upcoming changes.
>>
>> The exciting news is this: - oopses will be much more extensible -
>> a bson serializable dict is used, rather than the rfc822 format
>
> bson-serializable, or bson-serialized?

both. In process the definition of an oops is 'bson serializable
dict', and the new bson serializer for oops-datedir-repo uses bson.

I forgot to mention this also gets rid of the use of oops prefixes as
a way to make multiple writers safe - the hash naming means two
processes can share an oops dir and an oops prefix and not stomp on
each others oopses.

>> The downside, and the reason for this mail *before* enacting all
>> these changes, is that the change to bson will make OOPSes less
>> greppable on devpad.
>
> Oopses have a lot of text, so the standard unix tools work well with
> them.  Why not use json rather than bson, to retain that compatibility?

We have binary crud that arises in a couple of places - e.g. urls, I
think its worth not having to special handle that, rather than having
to have another layer escaping or encoding it. This pays off in
simpler code now, and in easier development of extensions down the
track : exactly what they will be I can't predict :). In the short
term its a nuisance, but in the longer term - we can't sensibly be
grepping for results in a 27 million document corpus (which is how
many oopses oops-tools knows about right now : many may have been
purged off disk, so this is symptomatic of the architecture being
inappropriately split / not synchronised).

We can extract specific fields from bson docs in a shell pipeline
(even still using grep to find the bson docs one is interested in) and
apply further text processing with ye old shell tools, so the
capability isn't /removed/, though it does become a little more
awkward.

-Rob