launchpad-reviewers team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

Stuart Bishop has proposed merging lp:~stub/launchpad/db-deploy into lp:launchpad.

Requested reviews:
  Stuart Bishop (stub)
Related bugs:
  Bug #906222 in Launchpad itself: "More BAD_USERS creating false positives during fast downtime updates"
  https://bugs.launchpad.net/launchpad/+bug/906222
  Bug #911417 in Launchpad itself: "Staging db restore fails on make -C database/replication stagingsetup"
  https://bugs.launchpad.net/launchpad/+bug/911417
  Bug #1025396 in Launchpad itself: "deployment scripts need to cope with streaming replication"
  https://bugs.launchpad.net/launchpad/+bug/1025396
  Bug #1025399 in Launchpad itself: "full-update.py should wait for streaming slaves before completing"
  https://bugs.launchpad.net/launchpad/+bug/1025399
  Bug #1036694 in Launchpad itself: "security.py still references slony"
  https://bugs.launchpad.net/launchpad/+bug/1036694

For more details, see:
https://code.launchpad.net/~stub/launchpad/db-deploy/+merge/127491

= Summary =

We are unnecessarily killing connections on slave databases during fast downtime database updates.

== Proposed fix ==

Don't do that.

== Pre-implementation notes ==

== LOC Rationale ==

== Implementation details ==

== Tests ==

== Demo and Q/A ==


= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  database/schema/preflight.py

./database/schema/preflight.py
      14: '_pythonpath' imported but unused
-- 
https://code.launchpad.net/~stub/launchpad/db-deploy/+merge/127491
Your team Launchpad code reviewers is subscribed to branch lp:launchpad.

=== modified file 'database/schema/preflight.py'
--- database/schema/preflight.py	2012-09-28 06:15:58 +0000
+++ database/schema/preflight.py	2012-10-02 13:58:38 +0000
@@ -255,7 +255,7 @@
                     max_lag = max(max_lag, lag)
             if max_lag < MAX_LAG:
                 break
-            time.sleep(0.2)
+            time.sleep(0.1)
 
         if max_lag < timedelta(0):
             streaming_lagged = False
@@ -331,19 +331,23 @@
     def check_open_connections(self):
         """Kill all non-system connections to Launchpad databases.
 
-        We only check on subscribed nodes, as there will be active systems
-        connected to other nodes in the replication cluster (such as the
-        SSO servers).
+        If replication is paused, only connections on the master database
+        are killed.
 
         System users are defined by SYSTEM_USERS.
         """
         # We keep trying to terminate connections every 0.5 seconds for
         # up to 10 seconds.
         num_tries = 20
-        seconds_to_pause = 0.5
+        seconds_to_pause = 0.1
+        if self.replication_paused:
+            nodes = set([self.lpmain_master_node])
+        else:
+            nodes = self.lpmain_nodes
+
         for loop_count in range(num_tries):
             all_clear = True
-            for node in self.lpmain_nodes:
+            for node in nodes:
                 cur = node.con.cursor()
                 cur.execute("""
                     SELECT