launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:mlist-sync-retry into launchpad:master.

Commit message:
Retry mlist-sync's call to updateTeamAddresses

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/376756

The new updateTeamAddresses API used by mlist-sync is a bit liable to time out when the database is cold.  Testing the relevant query by hand on staging and dogfood suggests that it warms up quite quickly on subsequent calls; so, rather than doing anything more elaborate for a staging-only API, just retry the call a few times if we get an OOPS.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:mlist-sync-retry into launchpad:master.

diff --git a/lib/lp/services/mailman/scripts/mlist_sync.py b/lib/lp/services/mailman/scripts/mlist_sync.py
index 2aa245e..67f8032 100644
--- a/lib/lp/services/mailman/scripts/mlist_sync.py
+++ b/lib/lp/services/mailman/scripts/mlist_sync.py
@@ -29,6 +29,7 @@ import os
 import subprocess
 import sys
 import textwrap
+import time
 
 from six.moves.xmlrpc_client import Fault
 
@@ -117,9 +118,22 @@ class MailingListSyncScript(LaunchpadScript):
         from Mailman.MailList import MailList
         from Mailman.Queue import XMLRPCRunner
 
-        # Ask Launchpad to update all the team email addresses.
+        # Ask Launchpad to update all the team email addresses.  This can be
+        # a bit timeout-prone if the relevant tables are cold, but should
+        # warm up reasonably quickly, so allow for a few retries.
         proxy = XMLRPCRunner.get_mailing_list_api_proxy()
-        proxy.updateTeamAddresses(self.options.hostname)
+        max_retries = 3
+        for i in range(max_retries + 1):
+            try:
+                proxy.updateTeamAddresses(self.options.hostname)
+            except Fault as fault:
+                if (fault.faultCode == faults.OopsOccurred.error_code and
+                        i < max_retries):
+                    self.logger.warning(
+                        "updateTeamAddresses OOPSed.  Retrying ...")
+                    time.sleep(2 ** i)
+                else:
+                    raise
 
         # Clean things up per mailing list.
         for list_name in Utils.list_names():