launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Sat, Apr 14, 2012 at 3:37 AM, Jonathan Lange <jml@xxxxxxxxx> wrote:
Bah, I forgot to address these.

> I'd rather raise ideas here in case because they probably need refinement:
>
>  * reduce test suite run time without reducing coverage

This would be nice. I think it's hard, short of the parallelisation
style work that Gary et al are doing at the moment, to have a good
developer time/result tradeoff. E.g. a week of developer time on
speedups giving a 30 minute improvement(a massive one!) would still
take 80 test runs to show a return. Good to do at small scale, but
hardware - once we parallelise - is /relatively/ cheap. We should
strive to keep it snappy, but time spent just making the existing
tests faster is hard to reason about. Perhaps this: timebox any
efforts in this direction.

How much of an improvement is worth 1K lines? I don't know. You could
say that an improvement equal to the time to run the tests for that
new 1K lines would offset the 1K lines entirely : and in terms of test
speed it would, but 1K lines also brings new defects, new concepts to
understand, new places for bugs...

I'd probably want a factor of 2 improvment to waive LoC on the basis
of test run time: if you add 1K LoC with tests taking 30 seconds, I'd
waive the LoC if you reduce the existing test run time by 60 seconds.

Francis may have a different answer  :)

>  * improve test suite coverage (I assume this bypasses LoC restrictions?)

Not at all. Coverage is good. LoC restrictions are good. Doing both is
excellent. We have a raft of slow multi-layer tests masquerading as
unit tests, we test some parts of the system thousands of times more
than other parts because of this. I'm sure you can have simpler
smaller targeted tests with better coverage in less LoC.

>  * something about making tests more focused

Meh. Doesn't, IME, make a significant difference to maintainability as
a first order effect. Second order effect sure - but theres a direct
driver for those (such as 'slow tests' or 'too much LoC'.).

-Rob