launchpad-reviewers team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

Robert Collins has proposed merging lp:~lifeless/python-oops-tools/prune into lp:python-oops-tools.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~lifeless/python-oops-tools/prune/+merge/84892

Pruning too many oopses at once makes django orm sad.
-- 
https://code.launchpad.net/~lifeless/python-oops-tools/prune/+merge/84892
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~lifeless/python-oops-tools/prune into lp:python-oops-tools.

=== modified file 'src/oopstools/oops/models.py'
--- src/oopstools/oops/models.py	2011-11-21 04:50:37 +0000
+++ src/oopstools/oops/models.py	2011-12-08 04:59:25 +0000
@@ -129,14 +129,20 @@
     def prune_unreferenced(klass, prune_from, prune_until, references):
         # XXX: This trusts date more than we may want to - should consider
         # having a date_received separate to the date generated.
-        to_delete = set(Oops.objects.filter(
-            date__gte=prune_from, date__lte=prune_until).exclude(
-            oopsid__in=references))
-        # deleting 1 at a time is a lot of commits, but leaves the DB lively
-        # for other transactions. May need to batch this in future if its
-        # too slow.
-        for oopsid in to_delete:
-            Oops.objects.filter(oopsid__exact=oopsid).delete()
+        while True:
+            # There may be very many OOPS to prune, so only ask for a few at a
+            # time. This prevents running out of memory in the prune process
+            # (can happen when millions of reports are being pruned at once).
+            to_delete = set(Oops.objects.filter(
+                date__gte=prune_from, date__lte=prune_until).exclude(
+                    oopsid__in=references)[:10000])
+            # deleting 1 at a time is a lot of commits, but leaves the DB lively
+            # for other transactions. May need to batch this in future if its
+            # too slow.
+            for oopsid in to_delete:
+                Oops.objects.filter(oopsid__exact=oopsid).delete()
+            if not to_delete:
+                break
 
 
 class Report(models.Model):