launchpad-dev team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx>]

For data points:

In the last successful build:


Page tests: 50m (~20%)
Bugs windmill: 10m (~4%)
Code windmill: 5m   (~2%)
Registry windmill: 7m (~3%)
Soyuz windmill: 1m (0%)
Translations windmill: 8m ((~3%)

Windmill + page tests: (~32%)
Total tests run time: 254m (100%)

So we are talking about a third of the test suite time taken up by the page 
and windmill tests.

-- 
Francis J. Lacoste
francis.lacoste@xxxxxxxxxxxxx

On November 3, 2010, Guilherme Salgado wrote:
> On Tue, 2010-11-02 at 08:59 -0500, Deryck Hodge wrote:
> > Hi, all.
> > 
> > I have some thoughts about our page and windmill tests, and I'd like
> > to propose an experiment related to these tests.  This is inspired by
> > my own thinking about how to best test web UI and by Rob's recent
> > cost/benefit analysis arguments for optional reviews.
> > 
> > Here's my assessment of the problem:
> > 
> > The burden we carry for our current page tests and Windmill tests does
> > not match the benefit we get from them.
> > 
> > The burden:
> > 
> > * Much longer test run times (the bugs module goes from 45 minutes to
> > 20 locally without them)
> > * Fragile tests that block landings
> > * Fragile infrastructure (see issues with Windmill tests under load)
> > * Confusion over how to best test UI (page tests vs. integration vs.
> > browser unit tests)
> 
> Although most of the above is true for page tests, I think it's a lot
> more problematic for windmill tests (specially the run time and fragile
> infrastructure), so it may be worth considering your experiment just for
> windmill tests?
> 
> I'd expect most of the 25 minutes saved by disabling page/windmill tests
> would be spent in the windmill ones, but maybe you already know how long
> it takes to run the tests under lp.app.bugs disabling just the windmill
> tests?

Attachment: signature.asc
Description: This is a digitally signed message part.