launchpad-dev team mailing list archive

[Brad Crittenden <brad.crittenden@xxxxxxxxxxxxx>]

On Nov 16, 2011, at 12:58 , Gary Poster wrote:

> Update:
> 
> Brad has looked into this, and I tried to do an assist.  We are still working on it.
> 
> If anyone is interested in the details:
> 
> - This can be reproduced reliably by running the full test suite on ec2 or buildbot.
> - We can't yet dupe the problem locally, running all the tests in the layer or all the tests in the pertinent module or all the tests in lp.code.
> - We've looked at broad diffs and have not seen a smoking gun.
> - The symptom has two oddities: it says that we are duplicating a factory generated ID, which should never happen, *and* it is the same "randomly generated" ID for every test run, which should (almost) never happen.
> 
> People report seeing this first in LP revision 14301.  Buildbot last passed revision 14297.  Therefore (given that this is not intermittent) the revisions that may have caused this are these:
> 
> 14298 Ian Booth [r=sinzui][bug=879103] Prevent teams from transitioning to an illegal
> 	subscription policy.
> 
> 14299 Stuart Bishop [r=rvb][no-qa] Attach a link to the previous OOPS generated in the
> request, if any.
> 
> 14300 Gavin Panella [r=rvb][bug=874250] Introduce an LRU cache for (distribution_id,
> sourcepackagename_id) --> dsp_id mappings to improve performance of
> the DistributionSourcePackage --> DistributionSourcePackageInDatabase
> smoke-and-mirrors trick.
> 
> 14301 Gavin Panella [r=rvb][bug=874250] Set max_bug_heat with a lazily evaluated
> expression in recalculateBugHeatCache().
> 


Code inspection of these four revisions didn't reveal anything suspicious.  At Gary's suggestion I created four branches, each that reverted one of the revisions.  Running through ec2 the branch reverting r14301 passed and the other three failed making it highly suspicious.

I have submitted the branch that reverts r 14301 (link to MP below).  Gavin this was your branch so you'll need to revisit it.

https://code.launchpad.net/~allenap/launchpad/bugnomination-timeout-bug-874250-lazy-heat/+merge/82260

--Brad


> Being unable to duplicate this outside of the full suite is a real drag, of course, but we're on the trail.
> 
> Gary
> 
> On Nov 16, 2011, at 11:37 AM, Jeroen Vermeulen wrote:
> 
>> On 2011-11-16 21:49, Gary Poster wrote:
>>> This looks like the same problem that currently has us in testfix.
>>> 
>>> https://lpbuildbot.canonical.com/builders/lucid_lp/builds/1559/steps/shell_6/logs/summary
>> 
>> That's a relief, because both of my EC2 tests (separate branches) came back with the same error.
>> 
>> 
>> Jeroen
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~launchpad-dev
>> Post to     : launchpad-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~launchpad-dev
>> More help   : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~launchpad-dev
> Post to     : launchpad-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~launchpad-dev
> More help   : https://help.launchpad.net/ListHelp

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail