launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Tue, Nov 2, 2010 at 12:05 PM, Gary Poster <gary.poster@xxxxxxxxxxxxx> wrote:
>
> On Nov 2, 2010, at 9:59 AM, Deryck Hodge wrote:
>
>> What do you all think?
>
> I think your argument has some merit, and I very much appreciate the reasoned way you approached it.
>
> Personally, though, I disagree with the proposal to throw the tests out.

Thanks for the feedback, Gary.  I understand this is a huge change,
and I appreciate this kind of response.

>
>  * Regression tests pay back over months and years, not suddenly.
>  * I am skeptical of the idea of "temporarily" or experimentally throwing them out.
>   - I believe that without maintenance, they will die.
>   - I don't believe that the proposed metrics ("regression" tags in bugs after tossing out tests for two or three months) for evaluating the experiment convey their value.

This echoes my own concern for Robert's proposal -- that it was hard
to verify value -- so I understand your feelings.  i.e. that we are
only verifying the absence of negative impact, not value of the thing
in question.

However, I think regression bugs will allow us to measure something
about the value of the tests, even if it is an incomplete picture.
This is the position I arrived at with Robert's proposal.  I'm also
skeptical that we can determine the pure value of review from his
experiment, but I think the data from that experiment will still be
useful.  I feel the same about my own proposal -- it's not a perfect
experiment, but it brings some value and at least gets us making an
assessment of the pain of parts of our test suite.

>  * While Windmill pain has been recurring, the proposal comes right now because of immediate pain that is caused by unusually large changes to the way tests are run.  Maris is trying to address that pain (while he simultaneously has other, more scheduled irons in the fire).
>
> Because we don't have a metric for measuring the tests' value, nor a way to quantitatively compare their value to their cost, this is a matter of opinion.  This is mine.
>

FWIW, while the pain of Windmill has been acute lately, I don't recall
a month without some form of Windmill pain since I've been working on
Launchpad.  I feel the same about page tests.

Again, thanks for the feedback.  I very much value your thoughts on this.

Cheers,
deryck


-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/