launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Wed, Nov 3, 2010 at 5:06 AM, Jonathan Lange <jml@xxxxxxxxxxxxx> wrote:
> On Wed, Nov 3, 2010 at 2:01 AM, Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx> wrote:
>> We don't know the value of these tests or how much usefulness they
>> provide.  We don't know the amount of coverage we actually get from
>> them.  We have half or better of a four+ hour test run for these tests
>> and we have no way to assess the value of this cost.  We have
>> developers avoiding page tests and windmill at all costs and no clear
>> story of even why we use them or what we expect from them.  It seems
>> to me a short and controlled experiment is an easy way to get some
>> data on these questions.
>>
>
> Would you not get the same data by running the page & windmill tests
> separately as Gary suggests?
>

Yes, you could certainly get data this way.  And as I said in Francis'
email, if people really prefer this approach, then I'll withdraw my
proposal.

I'm not personally interested in championing what Gary suggests, not
because I don't see some value over what we have now, but because I'm
just not as passionate about it nor do I feel the benefit is worth the
effort.  I'm skeptical that we'll ever make a determination about the
value or benefit of these tests once we set up another builder, mostly
because by setting up the builder there is an explicit assertion that
these tests have value and we shouldn't lose them.  So I imagine a
year from now these tests will be blocking deployments, and people
will be complaining the same as they do now for landings, but we'll
still be reluctant to change anything because we assume test failure
implies value.

Also, part of my goal with disabling page and windmill tests is to get
us looking at the UI more in QA.  If the tests are run before
deployment, there is no incentive to really examine the UI and user
interactions during QA.  The tests provide false assurance of UI
quality, and I don't see this changing if we set up another builder.

Cheers,
deryck


-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/