launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #05862
[Merge] lp:~lifeless/python-oops-tools/prune into lp:python-oops-tools
Robert Collins has proposed merging lp:~lifeless/python-oops-tools/prune into lp:python-oops-tools.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~lifeless/python-oops-tools/prune/+merge/84892
Pruning too many oopses at once makes django orm sad.
--
https://code.launchpad.net/~lifeless/python-oops-tools/prune/+merge/84892
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~lifeless/python-oops-tools/prune into lp:python-oops-tools.
=== modified file 'src/oopstools/oops/models.py'
--- src/oopstools/oops/models.py 2011-11-21 04:50:37 +0000
+++ src/oopstools/oops/models.py 2011-12-08 04:59:25 +0000
@@ -129,14 +129,20 @@
def prune_unreferenced(klass, prune_from, prune_until, references):
# XXX: This trusts date more than we may want to - should consider
# having a date_received separate to the date generated.
- to_delete = set(Oops.objects.filter(
- date__gte=prune_from, date__lte=prune_until).exclude(
- oopsid__in=references))
- # deleting 1 at a time is a lot of commits, but leaves the DB lively
- # for other transactions. May need to batch this in future if its
- # too slow.
- for oopsid in to_delete:
- Oops.objects.filter(oopsid__exact=oopsid).delete()
+ while True:
+ # There may be very many OOPS to prune, so only ask for a few at a
+ # time. This prevents running out of memory in the prune process
+ # (can happen when millions of reports are being pruned at once).
+ to_delete = set(Oops.objects.filter(
+ date__gte=prune_from, date__lte=prune_until).exclude(
+ oopsid__in=references)[:10000])
+ # deleting 1 at a time is a lot of commits, but leaves the DB lively
+ # for other transactions. May need to batch this in future if its
+ # too slow.
+ for oopsid in to_delete:
+ Oops.objects.filter(oopsid__exact=oopsid).delete()
+ if not to_delete:
+ break
class Report(models.Model):