launchpad-dev team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

On Sat, Oct 9, 2010 at 2:52 AM, Francis J. Lacoste
<francis.lacoste@xxxxxxxxxxxxx> wrote:
> After a Slony-1 restart and another attempt. It failed again but at a
> different place this time around:
>
> https://pastebin.canonical.com/38407/
>
> The warnings in Slony Node 5 log file are:
> lpslon_node5_node_staging.log:2010-10-08 19:31:49 UTCWARN
> remoteWorkerThread_4: transactions earlier than XID 16168895 are still in
> progress
> lpslon_node5_node_staging.log:2010-10-08 19:31:49 UTCWARN
> remoteWorkerThread_4: data copy for set 666 failed 174 times - sleep 60
> seconds

So this is a warning, not an error, and means what is says. Slony is
waiting for a long running transaction to complete before it can
proceed. This usually happens because we don't shut down the cronjobs
during the upgrade. We can use the new cronscript control file to do
this - I emailed the losas details on this just after it landed but it
hasn't been wired up yet.


> What this means, and how to recover from this... I have no idea.
>
> In the monkey-pushes-button-until-it-works fashion, I've asked Chex to restart
> Pg/Slony and re-attempt the restore...

Restarting slony will help if one of the daemons has fallen over.
Restarting PostgreSQL will terminate any rogue transactions blocking
the upgrade. The upgrade scripts already do the first to catch that
failure. We still need to shutdown cronscripts during the update and
terminate existing transactions (preferably without generating
spurious OOPS) - losas need to do the first task and I'll need to do
that second one.

It looks like the db patching has completed now:

Fri Oct 8 22:17:03 UTC 2010 Security applied to slave DB

The full database rebuild should kick in in a few hours.

I can't tell why Slony initially blocked - unfortunatly I used '>'
instead of '>>' and old logs are being trashed when we bounce the
daemons. Bug #657186 (opening a bug is overkill for a one character
fix, but I'm not on my dev box at the moment).

-- 
Stuart Bishop <stuart@xxxxxxxxxxxxxxxx>
http://www.stuartbishop.net/