launchpad-reviewers team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

Stuart Bishop has proposed merging lp:~stub/launchpad/trivial into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  Bug #951401 in Launchpad itself: "parse-ppa-apache-logs failing (missing files)"
  https://bugs.launchpad.net/launchpad/+bug/951401
  Bug #1263002 in Launchpad itself: "Twisted feature flag support fails typecasting when updating"
  https://bugs.launchpad.net/launchpad/+bug/1263002

For more details, see:
https://code.launchpad.net/~stub/launchpad/trivial/+merge/239178

Rather manually incrementing the 'start' LibrariFileContent in crontab to avoid backscans, calculate it by date.

The query is composed to do a backwards index scan, and is as performant as we can make it without an index on datecreated.

This will be irrelevant once we are removing files from disk, as removed files don't need to be iterated over and we can stop using the --start argument entirely.
-- 
https://code.launchpad.net/~stub/launchpad/trivial/+merge/239178
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~stub/launchpad/trivial into lp:launchpad.

=== modified file 'cronscripts/librarian-feed-swift.py'
--- cronscripts/librarian-feed-swift.py	2014-10-15 13:15:17 +0000
+++ cronscripts/librarian-feed-swift.py	2014-10-22 11:12:35 +0000
@@ -11,8 +11,10 @@
 
 import os
 
+from lp.services.database.interfaces import ISlaveStore
+from lp.services.librarian.model import LibraryFileContent
+from lp.services.librarianserver import swift
 from lp.services.scripts.base import LaunchpadCronScript
-from lp.services.librarianserver import swift
 
 
 class LibrarianFeedSwift(LaunchpadCronScript):
@@ -28,11 +30,21 @@
             dest="start", metavar="CONTENT_ID",
             help="Migrate files starting from CONTENT_ID")
         self.parser.add_option(
+            "--start-since", action="store", dest='start_since',
+            default=None, metavar="INTERVAL",
+            help="Migrate files starting from INTERVAL (PostgreSQL syntax)")
+        self.parser.add_option(
             "-e", "--end", action="store", type=int, default=None,
             dest="end", metavar="CONTENT_ID",
             help="Migrate files up to and including CONTENT_ID")
 
     def main(self):
+        if self.options.start_since:
+            self.options.start = ISlaveStore(LibraryFileContent).execute("""
+                SELECT MAX(id) FROM LibraryFileContent
+                WHERE datecreated < current_timestamp at time zone 'UTC'
+                    - CAST(%s AS INTERVAL)
+                """, (unicode(self.options.start_since),)).get_one()[0]
         if self.options.ids and (self.options.start or self.options.end):
             self.parser.error(
                 "Cannot specify both individual file(s) and range")