launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Tue, Nov 2, 2010 at 1:54 PM, Francis J. Lacoste
<francis.lacoste@xxxxxxxxxxxxx> wrote:
>
> On November 2, 2010, Deryck Hodge wrote:
>> On Tue, Nov 2, 2010 at 12:48 PM, Robert Collins
>>
>> <robert.collins@xxxxxxxxxxxxx> wrote:
>> > On Wed, Nov 3, 2010 at 6:44 AM, Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>
> wrote:
>> >> On Tue, Nov 2, 2010 at 12:33 PM, Robert Collins
>> >>
>> >> <robertc@xxxxxxxxxxxxxxxxx> wrote:
>> >>> 2) Whats the rollback strategy?
>> >>
>> >> I see this as "re-enable the tests and go back to what we have now."
>> >> I feel like that is too simple an answer for your question, so maybe
>> >> I'm missing something that your concerned about?
>> >
>> > Its just that tests tend to bitrot when they aren't run. It may be
>> > hard to turn them back on.
>>
>> Yeah, that's a valid concern.  I don't see anyway to avoid that risk.
>> I don't think 3 months is so long that the bitrot will be that huge
>> should this prove a failure, but that's just a gut feeling and I have
>> little to base that on, save how often page and windmill tests seem to
>> be changed normally.  Usually the changes are simple string
>> substitutions.  I, of course, would be willing to take on this pain
>> and fix the tests since I'm the one proposing the experiment.
>>
>
> I think making sure that the experiment is easy to revert is a very valid
> concern.
>
> I would suggest we do it differently then to ensure that possibility:
>
> Stop running pagetests/windmill pre-commit. Move them to a standalone buildbot
> that runs them exclusively, as much as it can. That buildbot blocks
> deployment, so no deployment of a stretch of QA revisions until a clean
> windmill/pagetests is blessed.
>
> I think this has several benefits:
>
> a- Ensure that the tests don't bitrot.
> b- Enforces the requirements of moving pagetest coverage into unit test
> coverage.
> c- Still speeds up the landing of code (altough probably doesn't speed up the
> deployment of code as much.)
> d- Makes the cost/benefit evaluation of the change more obvious:
>   - Classify the failures on the integration buildbot into:
>        - Instability failure
>        - Regression (failed windmill or page test)
>
> During the experiment, we shouldn't add any new windmill or pagetest. At the
> end of the evaluation period, we should look at the data in d, argue over the
> relative weight of loss of time due to instability failure vs regression and
> either: keep that new setup, revert to the old one, drop everything and move
> forward.

This feel like a different experiment than I'm proposing.  There's a
lot in that to save tests that *might* be useful, and for the course
of the experiment, doesn't leave us in a markedly different state than
we are today.  If everyone were standing up saying "these tests have
saved us time and again, what are you thinking?!?!" I could understand
this reaction.

At the point we start getting into alternate builders and testing
landings vs. testing deployments that gets into more than I can
reasonably champion, i.e. it's a foundations dev story not a process
experiment.  I'll happily make changes to the process I proposed if
there are changes I can make that keep this process driven but make
people feel better about my proposal, but I really don't want to take
on a build environment dev story.  Of course, if everyone feels better
about this alternate testing story, then I'm happy to defer to that
and drop the proposal.

Cheers,
deryck



-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/