launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

On Fri, 19 Nov 2010 01:16:26 +0000, Max Bowsher <maxb@xxxxxxx> wrote:
> It would appear that, as far as I can see, no code imports have
> successfully completed since some time on 2010-11-15.
> 
> The code import machines are all full of running jobs, all of which
> appear to sit around doing nothing much, until they get canceled an hour
> after they start.
> 
> Because every dispatched importd job is now taking 60 minutes, which is
> much longer than the average when things work properly, there are now
> *very many* imports which are queuing for an importd execution, hence
> the web UI is displaying "The next import is scheduled to run as soon as
> possible." for imports which won't actually be attempted for hours?
> days? (and will then fail.)

I've got a few more bits of information to add to this.  The logs look
like this:

2010-11-18 22:35:30 INFO    [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
2010-11-18 22:35:32 INFO    [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
Exception KeyboardInterrupt: KeyboardInterrupt() in <function terminate at 0x9ba86bc> ignored
2010-11-18 23:35:35 INFO    [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
2010-11-18 23:35:35 INFO    [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.

The "Opened sftp connection" log lines are new-ish, but they have been
present for at least a few weeks, so they are not that closely related
to the issue.

The issue appears on staging too, which suggests to me that it is more
likely to be a code change than an environmental one.

The importd user on the importd slaves can still sftp to the central
store, at least in a trivial way.

Although the problem appeared soon after the dustup with the XML-RPC
service over the weekend, it doesn't actually seem to be related: there
were some successful imports after all that drama.

There is no LOSA around today, which makes finding more information
hard.

Now some guesswork.

There was a nodowntime rollout on the 15th.  I bet it introduced the
problem.

My utter WAG is that it was the upgrade to bzr 2.2.1 that caused the
problem.

Cheers,
mwh