launchpad-reviewers team mailing list archive

[Colin Watson <mp+425836@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:mirror-prober-recursion into launchpad:master.

Commit message:
Defer mirror probes to the next tick of the reactor

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  Bug #1975824 in Launchpad itself: "mirror not listed on launchpad.net"
  https://bugs.launchpad.net/launchpad/+bug/1975824

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/425836

Otherwise we end up with excessively deep recursion, and Python eventually crashes.

I wasn't entirely sure about this, but we applied it temporarily to production and it seems to have fixed the mirror prober crashes that have been plaguing us for some time (touch wood).
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:mirror-prober-recursion into launchpad:master.

diff --git a/lib/lp/registry/scripts/distributionmirror_prober.py b/lib/lp/registry/scripts/distributionmirror_prober.py
index 965b8a6..281fb8c 100644
--- a/lib/lp/registry/scripts/distributionmirror_prober.py
+++ b/lib/lp/registry/scripts/distributionmirror_prober.py
@@ -36,6 +36,7 @@ from twisted.internet.defer import (
     DeferredSemaphore,
     )
 from twisted.internet.ssl import VerificationError
+from twisted.internet.task import deferLater
 from twisted.python.failure import Failure
 from twisted.web.client import (
     Agent,
@@ -119,7 +120,14 @@ class RequestManager:
                 self.overall_semaphore,
                 DeferredSemaphore(self.max_parallel_per_host))
             self.host_locks[host] = multi_lock
-        return multi_lock.run(probe_func)
+
+        def probe():
+            if reactor.running:
+                return deferLater(reactor, 0, probe_func)
+            else:
+                return probe_func()
+
+        return multi_lock.run(probe)
 
 
 class MultiLock(defer._ConcurrencyPrimitive):