launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Wed, Nov 3, 2010 at 9:15 AM, Guilherme Salgado <salgado@xxxxxxxxxxxxx> wrote:
> On Tue, 2010-11-02 at 08:59 -0500, Deryck Hodge wrote:
>> Hi, all.
>>
>> I have some thoughts about our page and windmill tests, and I'd like
>> to propose an experiment related to these tests.  This is inspired by
>> my own thinking about how to best test web UI and by Rob's recent
>> cost/benefit analysis arguments for optional reviews.
>>
>> Here's my assessment of the problem:
>>
>> The burden we carry for our current page tests and Windmill tests does
>> not match the benefit we get from them.
>>
>> The burden:
>>
>> * Much longer test run times (the bugs module goes from 45 minutes to
>> 20 locally without them)
>> * Fragile tests that block landings
>> * Fragile infrastructure (see issues with Windmill tests under load)
>> * Confusion over how to best test UI (page tests vs. integration vs.
>> browser unit tests)
>
> Although most of the above is true for page tests, I think it's a lot
> more problematic for windmill tests (specially the run time and fragile
> infrastructure), so it may be worth considering your experiment just for
> windmill tests?
>
> I'd expect most of the 25 minutes saved by disabling page/windmill tests
> would be spent in the windmill ones, but maybe you already know how long
> it takes to run the tests under lp.app.bugs disabling just the windmill
> tests?
>

I'm actually not sure for the bugs module, but can do a run and see.
Francis told me moments ago that page tests run 45 minutes on the
builder and windmill lasts 30 minutes.  So more like 1/3 of total run
time for the entire test suite when times are combined.  Bugs may be
more of an offender here since we have done less work to move off page
tests and have 2-3 really long windmill tests.

Cheers,
deryck


-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/