launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

See bug 794802 for some of the gory details; in short bugsummary
managed to spring a nasty surprise on us.

We decided to roll forward given a few factors - the problem is
isolated (though to a substantial fraction of the system), its tricky
to change live no matter what, and we didn't know what the precise
cause was - without know that changing stuff in a panic wouldn't
necessarily fix it.. because its a schema change its not simple to
rollback. We didn't want to make things worse by poking haphazardly
without Stuarts help either, (so we got that :))

Much of the delay in fixing was our 4 hour test suite validating the
key change to fix the dirty-great-lock; thats fixed (Thanks stub), but
there was/is a hidden issue behind that - the UPDATE was doing table
scans due to the use of IS NOT DISTINCT FROM, which made the
implementation short and sweet but also made the query pathological.
So we easily added some seconds or more (at 0.3 seconds per dimension
combination) to bug updates, and with bug heat updates in Ubuntu were
slow anyway.

At this point I've handed over the incident to Stuart, and am halting() myself.

I hope to awaken to great news :)

-Rob