launchpad-reviewers team mailing list archive

[Colin Watson <mp+407273@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:librarian-iterate-cursor into launchpad:master.

Commit message:
librarian-gc: Iterate over cursors when deleting unwanted files

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/407273

Fetching rows from a named cursor using `fetchone` fetches one record at a time from the backend, which is very slow.  `psycopg2` has machinery to aggregate fetches, but to use that we need to iterate over the cursor instead.  See https://www.psycopg.org/docs/usage.html#server-side-cursors.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:librarian-iterate-cursor into launchpad:master.

diff --git a/lib/lp/services/librarianserver/librariangc.py b/lib/lp/services/librarianserver/librariangc.py
index 936ef3a..1df3fb6 100644
--- a/lib/lp/services/librarianserver/librariangc.py
+++ b/lib/lp/services/librarianserver/librariangc.py
@@ -647,10 +647,12 @@ def delete_unwanted_disk_files(con):
     cur.execute("""
         SELECT id FROM LibraryFileContent ORDER BY id
         """)
+    content_id_iter = iter(cur)
 
     def get_next_wanted_content_id():
-        result = cur.fetchone()
-        if result is None:
+        try:
+            result = next(content_id_iter)
+        except StopIteration:
             return None
         else:
             return result[0]
@@ -819,10 +821,12 @@ def delete_unwanted_swift_files(con):
     cur.execute("""
         SELECT id FROM LibraryFileContent ORDER BY id
         """)
+    content_id_iter = iter(cur)
 
     def get_next_wanted_content_id():
-        result = cur.fetchone()
-        if result is None:
+        try:
+            result = next(content_id_iter)
+        except StopIteration:
             return None
         else:
             return result[0]
diff --git a/lib/lp/testing/pgsql.py b/lib/lp/testing/pgsql.py
index 373b021..2aa623f 100644
--- a/lib/lp/testing/pgsql.py
+++ b/lib/lp/testing/pgsql.py
@@ -126,6 +126,9 @@ class CursorWrapper:
         else:
             return setattr(self.real_cursor, key, val)
 
+    def __iter__(self):
+        return iter(self.real_cursor)
+
 
 _org_connect = None