launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

On Nov 2, 2010, at 9:59 AM, Deryck Hodge wrote:

> What do you all think?

I think your argument has some merit, and I very much appreciate the reasoned way you approached it.  

Personally, though, I disagree with the proposal to throw the tests out.

 * Regression tests pay back over months and years, not suddenly.
 * I am skeptical of the idea of "temporarily" or experimentally throwing them out.
   - I believe that without maintenance, they will die.
   - I don't believe that the proposed metrics ("regression" tags in bugs after tossing out tests for two or three months) for evaluating the experiment convey their value.
 * While Windmill pain has been recurring, the proposal comes right now because of immediate pain that is caused by unusually large changes to the way tests are run.  Maris is trying to address that pain (while he simultaneously has other, more scheduled irons in the fire).

Because we don't have a metric for measuring the tests' value, nor a way to quantitatively compare their value to their cost, this is a matter of opinion.  This is mine.

I've wanted to put Windmill on buildbot/Hudson, separately from the test suite and more as something that blocks rolling out, rather than landing.  Perhaps running them in isolation like that would reduce their fragility.  I don't want the results to be ignored.

Gary