launchpad-dev team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx>]

On November 2, 2010, Deryck Hodge wrote:
> On Tue, Nov 2, 2010 at 12:48 PM, Robert Collins
> 
> <robert.collins@xxxxxxxxxxxxx> wrote:
> > On Wed, Nov 3, 2010 at 6:44 AM, Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx> 
wrote:
> >> On Tue, Nov 2, 2010 at 12:33 PM, Robert Collins
> >> 
> >> <robertc@xxxxxxxxxxxxxxxxx> wrote:
> >>> 2) Whats the rollback strategy?
> >> 
> >> I see this as "re-enable the tests and go back to what we have now."
> >> I feel like that is too simple an answer for your question, so maybe
> >> I'm missing something that your concerned about?
> > 
> > Its just that tests tend to bitrot when they aren't run. It may be
> > hard to turn them back on.
> 
> Yeah, that's a valid concern.  I don't see anyway to avoid that risk.
> I don't think 3 months is so long that the bitrot will be that huge
> should this prove a failure, but that's just a gut feeling and I have
> little to base that on, save how often page and windmill tests seem to
> be changed normally.  Usually the changes are simple string
> substitutions.  I, of course, would be willing to take on this pain
> and fix the tests since I'm the one proposing the experiment.
> 

I think making sure that the experiment is easy to revert is a very valid 
concern.

I would suggest we do it differently then to ensure that possibility:

Stop running pagetests/windmill pre-commit. Move them to a standalone buildbot 
that runs them exclusively, as much as it can. That buildbot blocks 
deployment, so no deployment of a stretch of QA revisions until a clean 
windmill/pagetests is blessed.

I think this has several benefits:

a- Ensure that the tests don't bitrot.
b- Enforces the requirements of moving pagetest coverage into unit test 
coverage.
c- Still speeds up the landing of code (altough probably doesn't speed up the 
deployment of code as much.)
d- Makes the cost/benefit evaluation of the change more obvious:
   - Classify the failures on the integration buildbot into:
	- Instability failure
 	- Regression (failed windmill or page test)

During the experiment, we shouldn't add any new windmill or pagetest. At the 
end of the evaluation period, we should look at the data in d, argue over the 
relative weight of loss of time due to instability failure vs regression and 
either: keep that new setup, revert to the old one, drop everything and move 
forward.

Cheers
-- 
Francis J. Lacoste
francis.lacoste@xxxxxxxxxxxxx

Attachment: signature.asc
Description: This is a digitally signed message part.