launchpad-reviewers team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

Stuart Bishop has proposed merging lp:~stub/launchpad/feed-end-at into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~stub/launchpad/feed-end-at/+merge/259378

Add an option to librarian-feed-swift.py allowing us to not process newer files.

The goal is to ensure that we have at least two copies of Librarian data in two different systems (disk & swift, or swift & offsite backup). By running two jobs in sequence, the first moving all disk files into Swift older than $cutoff, and the second copying all disk files into Swift younger than $cutoff, we create a window where data exists both on disk and in Swift. Our other systems can ensure that the data is mirrored from Swift to offsite during that window.

The alternative approach would be a single run, and providing a new option to not remove files younger than $cutoff. This would be better, but less of a quick and easy fix.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~stub/launchpad/feed-end-at into lp:launchpad.

=== modified file 'cronscripts/librarian-feed-swift.py'
--- cronscripts/librarian-feed-swift.py	2014-12-16 09:20:15 +0000
+++ cronscripts/librarian-feed-swift.py	2015-05-18 13:30:47 +0000
@@ -35,11 +35,16 @@
         self.parser.add_option(
             "--start-since", action="store", dest='start_since',
             default=None, metavar="INTERVAL",
-            help="Migrate files starting from INTERVAL (PostgreSQL syntax)")
+            help="Migrate files older than INTERVAL (PostgreSQL syntax)")
         self.parser.add_option(
             "-e", "--end", action="store", type=int, default=None,
             dest="end", metavar="CONTENT_ID",
             help="Migrate files up to and including CONTENT_ID")
+        self.parser.add_option(
+            "--end-at", action="store", dest='end_at',
+            default=None, metavar="INTERVAL",
+            help="Don't migrate files older than INTERVAL "
+                 "(PostgreSQL syntax)")
 
     def main(self):
         if self.options.rename and self.options.remove:
@@ -58,6 +63,13 @@
                     - CAST(%s AS INTERVAL)
                 """, (unicode(self.options.start_since),)).get_one()[0]
 
+        if self.options.end_at:
+            self.options.end = ISlaveStore(LibraryFileContent).execute("""
+                SELECT MAX(id) FROM LibraryFileContent
+                WHERE datecreated < current_timestamp at time zone 'UTC'
+                    - CAST(%s AS INTERVAL)
+                """, (unicode(self.options.end_at),)).get_one()[0]
+
         if self.options.ids and (self.options.start or self.options.end):
             self.parser.error(
                 "Cannot specify both individual file(s) and range")