launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #06260
[Merge] lp:~wgrant/launchpad/stop-this-aging-nonsense into lp:launchpad
William Grant has proposed merging lp:~wgrant/launchpad/stop-this-aging-nonsense into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #505850 in Launchpad itself: "IBugSet.dangerousGetAllBugs() shouldn't exist"
https://bugs.launchpad.net/launchpad/+bug/505850
Bug #906193 in Launchpad itself: "BugHeatUpdater never completes"
https://bugs.launchpad.net/launchpad/+bug/906193
Bug #915269 in Launchpad itself: "bug heat aging compromises search performance"
https://bugs.launchpad.net/launchpad/+bug/915269
For more details, see:
https://code.launchpad.net/~wgrant/launchpad/stop-this-aging-nonsense/+merge/91763
08:54:33 < lifeless> wgrant: hey
08:54:40 < lifeless> wgrant: whats the status of the heat plumbing
08:59:26 < wgrant> lifeless: I was planning on waiting for a few days of insufficient complaints before ripping it out.
09:00:19 < lifeless> wgrant: doit
09:00:39 < lifeless> or at least, lets get the branch ready to troll
09:02:07 < wgrant> lifeless: Disable the max heat update code, and the aging job?
09:02:12 < wgrant> s/Disable/Delete/
09:02:24 < lifeless> yeah, and remove the flag
This branch removes the aging job. There's still aging code in calculate_bug_heat() in the DB, but that is separately exorcised by lp:~wgrant/launchpad/stop-this-aging-nonsense-db.
--
https://code.launchpad.net/~wgrant/launchpad/stop-this-aging-nonsense/+merge/91763
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/stop-this-aging-nonsense into lp:launchpad.
=== modified file 'lib/lp/bugs/doc/bug-heat.txt'
--- lib/lp/bugs/doc/bug-heat.txt 2011-09-30 16:05:20 +0000
+++ lib/lp/bugs/doc/bug-heat.txt 2012-02-07 05:58:19 +0000
@@ -200,108 +200,6 @@
>>> flags.cleanUp()
-Getting bugs whose heat is outdated
------------------------------------
-
-It's possible to get the set of bugs whose heat hasn't been updated for
-a given amount of time by calling IBugSet's getBugsWithOutdatedHeat()
-method.
-
-First, we'll set the heat of all bugs so that none of them are out of
-date.
-
- >>> from lp.bugs.interfaces.bug import IBugSet
- >>> for bug in getUtility(IBugSet).dangerousGetAllBugs():
- ... bug.setHeat(0)
-
-If we call getBugsWithOutdatedHeat() now, the set that is returned will
-be empty because all the bugs have been recently updated.
-getBugsWithOutdatedHeat() takes a single parameter, max_heat_age, which
-is the maximum age, in days, that a bug's heat can be before it gets
-included in the returned set.
-
- >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
- 0
-
-IBug.setHeat() takes a timestamp parameter so that we can set the
-heat_last_updated date manually for the purposes of testing. If we make
-a bug's heat older than the max_heat_age that we pass to
-getBugsWithOutdatedHeat() it will appear in the set returned by
-getBugsWithOutdatedHeat().
-
- >>> old_heat_bug = factory.makeBug()
- >>> old_heat_bug.setHeat(
- ... 0, datetime.now(timezone('UTC')) - timedelta(days=2))
-
- >>> outdated_bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(1)
- >>> outdated_bugs.count()
- 1
-
- >>> outdated_bugs[0] == old_heat_bug
- True
-
-getBugsWithOutdatedHeat() also returns bugs whose heat has never been
-updated.
-
- >>> new_bug = factory.makeBug()
-
-We'll set the new bug's heat_last_updated to None manually.
-
- >>> new_bug.heat_last_updated = None
-
- >>> outdated_bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(1)
- >>> outdated_bugs.count()
- 2
-
- >>> new_bug in outdated_bugs
- True
-
-
-The BugHeatUpdater class
----------------------------
-
-The BugHeatUpdater class is used to create bug heat calculation jobs for
-bugs with out-of-date heat.
-
- >>> from lp.scripts.garbo import BugHeatUpdater
- >>> from lp.services.log.logger import FakeLogger
-
-We'll commit the transaction so that the BugHeatUpdater updates the
-right bugs.
-
- >>> transaction.commit()
- >>> update_bug_heat = BugHeatUpdater(FakeLogger(), max_heat_age=1)
-
-BugHeatUpdater implements ITunableLoop and as such is callable. Calling
-it as a method will recalculate the heat for all the out-of-date bugs.
-
-There are two bugs with heat more than a day old:
-
- >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
- 2
-
-Calling our BugHeatUpdater will update the heat of those bugs.
-
- >>> update_bug_heat(chunk_size=1)
- DEBUG Updating heat for 1 bugs
-
-IBugSet.getBugsWithOutdatedHeat() will now return 1 item.
-
- >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
- 1
-
-Update the rest in one big chunk.
-
- >>> update_bug_heat(chunk_size=1000)
- DEBUG Updating heat for 1 bugs
-
-IBugSet.getBugsWithOutdatedHeat() will now return an empty set since all
-the bugs have been updated.
-
- >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
- 0
-
-
Caculating the maximum heat for a target
----------------------------------------
=== modified file 'lib/lp/bugs/interfaces/bug.py'
--- lib/lp/bugs/interfaces/bug.py 2012-01-03 10:34:20 +0000
+++ lib/lp/bugs/interfaces/bug.py 2012-02-07 05:58:19 +0000
@@ -1254,27 +1254,6 @@
Otherwise, return False.
"""
- def dangerousGetAllBugs():
- # XXX 2010-01-08 gmb bug=505850:
- # Note, this method should go away when we have a proper
- # permissions system for scripts.
- """DO NOT CALL THIS METHOD.
-
- This method exists solely to allow the bug heat script to grab
- all the bugs in the database - including private ones - and
- iterate over them. DO NOT USE IT UNLESS YOU KNOW WHAT YOU'RE
- DOING. AND IF YOU KNOW WHAT YOU'RE DOING YOU KNOW BETTER THAN TO
- USE THIS ANYWAY.
- """
-
- def getBugsWithOutdatedHeat(max_heat_age):
- """Return the set of bugs whose heat is out of date.
-
- :param max_heat_age: The maximum age, in days, that a bug's heat
- can be before it is included in the
- returned set.
- """
-
class IFileBugData(Interface):
"""A class containing extra data to be used when filing a bug."""
=== modified file 'lib/lp/bugs/model/bug.py'
--- lib/lp/bugs/model/bug.py 2012-01-04 17:41:01 +0000
+++ lib/lp/bugs/model/bug.py 2012-02-07 05:58:19 +0000
@@ -22,10 +22,6 @@
from cStringIO import StringIO
-from datetime import (
- datetime,
- timedelta,
- )
from email.Utils import make_msgid
from functools import wraps
from itertools import chain
@@ -39,7 +35,6 @@
)
from lazr.lifecycle.snapshot import Snapshot
import pytz
-from pytz import timezone
from sqlobject import (
BoolCol,
ForeignKey,
@@ -3025,25 +3020,6 @@
result_set = store.find(Bug, Bug.id.is_in(bug_numbers))
return result_set.order_by('id')
- def dangerousGetAllBugs(self):
- """See `IBugSet`."""
- store = IStore(Bug)
- result_set = store.find(Bug)
- return result_set.order_by('id')
-
- def getBugsWithOutdatedHeat(self, max_heat_age):
- """See `IBugSet`."""
- store = IStore(Bug)
- last_updated_cutoff = (
- datetime.now(timezone('UTC')) -
- timedelta(days=max_heat_age))
- last_updated_clause = Or(
- Bug.heat_last_updated < last_updated_cutoff,
- Bug.heat_last_updated == None)
-
- return store.find(
- Bug, Bug.duplicateof == None, last_updated_clause).order_by('id')
-
class BugAffectsPerson(SQLBase):
"""A bug is marked as affecting a user."""
=== modified file 'lib/lp/scripts/garbo.py'
--- lib/lp/scripts/garbo.py 2012-01-18 05:41:43 +0000
+++ lib/lp/scripts/garbo.py 2012-02-07 05:58:19 +0000
@@ -33,11 +33,8 @@
)
import transaction
from zope.component import getUtility
-from zope.security.proxy import removeSecurityProxy
from lp.answers.model.answercontact import AnswerContact
-from lp.bugs.interfaces.bug import IBugSet
-from lp.bugs.model.bug import Bug
from lp.bugs.model.bugattachment import BugAttachment
from lp.bugs.model.bugnotification import BugNotification
from lp.bugs.model.bugwatch import BugWatchActivity
@@ -56,7 +53,6 @@
from lp.registry.model.person import Person
from lp.services.config import config
from lp.services.database import postgresql
-from lp.services.database.constants import UTC_NOW
from lp.services.database.lpstorm import IMasterStore
from lp.services.database.sqlbase import (
cursor,
@@ -771,56 +767,6 @@
"""
-class BugHeatUpdater(TunableLoop):
- """A `TunableLoop` for bug heat calculations."""
-
- maximum_chunk_size = 5000
-
- def __init__(self, log, abort_time=None, max_heat_age=None):
- super(BugHeatUpdater, self).__init__(log, abort_time)
- self.transaction = transaction
- self.total_processed = 0
- self.is_done = False
- self.offset = 0
- if max_heat_age is None:
- max_heat_age = config.calculate_bug_heat.max_heat_age
- self.max_heat_age = max_heat_age
-
- self.store = IMasterStore(Bug)
-
- @property
- def _outdated_bugs(self):
- outdated_bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(
- self.max_heat_age)
- # We remove the security proxy so that we can access the set()
- # method of the result set.
- return removeSecurityProxy(outdated_bugs)
-
- def isDone(self):
- """See `ITunableLoop`."""
- # When the main loop has no more Bugs to process it sets
- # offset to None. Until then, it always has a numerical
- # value.
- return self._outdated_bugs.is_empty()
-
- def __call__(self, chunk_size):
- """Retrieve a batch of Bugs and update their heat.
-
- See `ITunableLoop`.
- """
- chunk_size = int(chunk_size + 0.5)
- outdated_bugs = self._outdated_bugs[:chunk_size]
- # We don't use outdated_bugs.set() here to work around
- # Storm Bug #820290.
- outdated_bug_ids = [bug.id for bug in outdated_bugs]
- self.log.debug("Updating heat for %s bugs", len(outdated_bug_ids))
- IMasterStore(Bug).find(
- Bug, Bug.id.is_in(outdated_bug_ids)).set(
- heat=SQL('calculate_bug_heat(Bug.id)'),
- heat_last_updated=UTC_NOW)
- transaction.commit()
-
-
class BugWatchActivityPruner(BulkPruner):
"""A TunableLoop to prune BugWatchActivity entries."""
target_table_class = BugWatchActivity
@@ -1226,7 +1172,6 @@
BugWatchScheduler,
UnusedSessionPruner,
DuplicateSessionPruner,
- BugHeatUpdater,
]
experimental_tunable_loops = []
