launchpad-dev team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

On Fri, Jun 10, 2011 at 4:19 PM, Robert Collins
<robertc@xxxxxxxxxxxxxxxxx> wrote:
> On Thu, Jun 9, 2011 at 10:44 PM, Robert Collins
> <robertc@xxxxxxxxxxxxxxxxx> wrote:
>> See bug 794802 for some of the gory details; in short bugsummary
>> managed to spring a nasty surprise on us.
> ...
>> At this point I've handed over the incident to Stuart, and am halting() myself.
>>
>> I hope to awaken to great news :)
>
> So this is basically behind us - there is a cowboyed schema change
> live that preserves performance - the branch for it to exist
> officially will be landing on devel soon. Until that lands the tags
> portlets for projects won't adjust (specifically closing all the bugs
> for a tag won't remove that tag).
>
> Some observations:
>  - we had a 900 timeout spike over the period of degraded service.
> Thats not too bad - its just over one every 2 minutes.

I think this will include some timeout spikes on unrelated pages.

Applying DB patches to live and busy systems is a learning experience.
Yesterday I learned that for some reason I don't yet understand,
creating an empty table with foreign key references requires locks on
the targeted table that conflicts with normal read/write operations,
and that lock can never be fully gained when the table is always being
read by at least one transaction.


-- 
Stuart Bishop <stuart@xxxxxxxxxxxxxxxx>
http://www.stuartbishop.net/