launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Jonathan Lange wrote:
> On Tue, Sep 29, 2009 at 7:49 AM, Michael Hudson
> <michael.hudson@xxxxxxxxxxxxx> wrote:
>> Michael Hudson wrote:
>>> Michael Hudson wrote:
>>>> Tim Penhey wrote:
>>>>> Hi Michael,
>>>>>
>>>>> We've had a few issues with the branch puller today.  It seems to get itself
>>>>> wedged, where it has no workers, but the main script doesn't die.
>>>> This has happened at least a few times today.
>>>>
>>>>> We restarted today and it worked but we don't know why it stopped working.
>>>> I still don't really have a clue.  The logging in the branch you landed
>>>> and one of mine below should give us a better idea what's going on.
>>>>
>>>>> jml added to the logging, but couldn't get it working on his karmic laptop so
>>>>> I committed it:
>>>>>     lp:~thumper/launchpad/fix-puller-logging
>>>>>
>>>>> And I'm running it through ec2 now with -s.
>>>> It landed fine.
>>>>
>>>>> Here are some others for you to look at :-)
>>>>>
>>>>> https://bugs.edge.launchpad.net/launchpad-code/+bug/438287
>>>> https://code.edge.launchpad.net/~mwhudson/launchpad/requestMirror-shouldnt-demote-branch/+merge/12561
>>>>
>>>>> https://bugs.edge.launchpad.net/launchpad-code/+bug/438290
>>>> https://code.edge.launchpad.net/~mwhudson/launchpad/puller-more-useful-xmlrpc-logs/+merge/12562
>> Oh, and this one has been cowboyed into production.  The puller has
>> fallen over once since the cowboy, and the log didn't provide any real
>> clues :/  Although this log output is a lot more informative than the old!
>>
> 
> I gather we still don't know what was going on then.

Correct.  As predicted by spm, now we have sufficient logging and
monitoring in place, the system is now much more reliable:

https://lpstats.canonical.com/graphs/BranchPullerRequestsAndDelay/

One outage in the last 5 days.

Cheers,
mwh