launchpad-dev team mailing list archive

[Curtis Hovey <curtis.hovey@xxxxxxxxxxxxx>]

Thank you for starting this conversation.

On Tue, 2010-11-02 at 08:59 -0500, Deryck Hodge wrote:
> I'd like to propose an experiment to see if this holds true.  I would
> propose that for 3 months (or until the Thunderdome) that we:
> 
> 1.) disable page tests and windmill tests
> 2.) leave windmill around for running yuitests and API integration
> tests
> 3.) anytime we touch UI in this time we must add unit test coverage
> for browser classes (if the page test was acting as a unit test for
> the view)
> 4.) we track closely any regression that slips through and the impact
> of this regression

I am +1 for the test.

> A note about #2, #3, and #4 here:
> 
> For #2, I think we should still run the JS unit tests automatically
> and in my experience Windmill is not fragile for yui tests.  Migrating
> to jstestdriver, if we decide post-experiement to abandon Windmill
> completely, should be a different issue, IMHO.  Francis noted
> yesterday that the API tests for the js LP.client have no other
> coverage, so we should leave those around as well.  Again, I don't
> think these change much or are subject to the same fragility.

I agree that the YUI test harness has never failed, but It will always
be possible since there is exactly 1 page call though windmill for each
registered suite.

> My assumption about #3 is that we have tested some bits in page tests
> that have no other test coverage.  For example, view methods that
> should be tested in browser unit tests but are covered by the page
> test.  I'm proposing we disable these tests but leave them around and
> that we add unit test coverage where appropriate when touching UI.  I
> am not suggesting we have browser unit tests attempt to replicate
> story tests.  I think it would be could to run the page tests locally
> when working on UI to see if something needs unit test coverage, too.
> But this can't really be enforced by the experiment.

I always delete any page test that contradicts a unittest. I have seen
page tests that are real stories (a user acceptance test); they only
fail when a feature is dramatically changed and often require a total
re-write.

I still favour assigning a several engineers to purge pagetests/stories
of duplication. I cringe every time I see a page test check for form
failures and permission denied.

-- 
__Curtis C. Hovey_________
http://launchpad.net/

Attachment: signature.asc
Description: This is a digitally signed message part