launchpad-dev team mailing list archive

[Guilherme Salgado <salgado@xxxxxxxxxxxxx>]

On Tue, 2010-11-02 at 08:59 -0500, Deryck Hodge wrote:
> Hi, all.
> 
> I have some thoughts about our page and windmill tests, and I'd like
> to propose an experiment related to these tests.  This is inspired by
> my own thinking about how to best test web UI and by Rob's recent
> cost/benefit analysis arguments for optional reviews.
> 
> Here's my assessment of the problem:
> 
> The burden we carry for our current page tests and Windmill tests does
> not match the benefit we get from them.
> 
> The burden:
> 
> * Much longer test run times (the bugs module goes from 45 minutes to
> 20 locally without them)
> * Fragile tests that block landings
> * Fragile infrastructure (see issues with Windmill tests under load)
> * Confusion over how to best test UI (page tests vs. integration vs.
> browser unit tests)

Although most of the above is true for page tests, I think it's a lot
more problematic for windmill tests (specially the run time and fragile
infrastructure), so it may be worth considering your experiment just for
windmill tests?

I'd expect most of the 25 minutes saved by disabling page/windmill tests
would be spent in the windmill ones, but maybe you already know how long
it takes to run the tests under lp.app.bugs disabling just the windmill
tests?

-- 
Guilherme Salgado <https://launchpad.net/~salgado>

Attachment: signature.asc
Description: This is a digitally signed message part