launchpad-dev team mailing list archive

[Paul Hummer <paul.hummer@xxxxxxxxxxxxx>]

On 11/02/2010 07:59 AM, Deryck Hodge wrote:
> Since we're moving to continuous rollouts and we're having to focus
> more on daily QA, any regression these tests would catch should be
> caught in QA.  If problems exist, we rollback the rev and try again.
> If not, we rollback the rev as soon as we catch it.  We can rely on
> ourselves and beta users to catch these regressions, and by not
> carrying the burden of the tests we get quicker cycles and landings.
> I'm not suggesting manual testing is better.  In a perfect world, we
> would have a fast test suite and get the benefit of both, but for now,
> the test bloat is causing more harm than the benefit it brings.  I
> think we could trust our QA process and users until we can get a
> sufficiently fast test suite to worry about web ui tests again.  All
> IMHO, of course. :-)
>
> I'd like to propose an experiment to see if this holds true.  I would
> propose that for 3 months (or until the Thunderdome) that we:
>
> 1.) disable page tests and windmill tests
> 2.) leave windmill around for running yuitests and API integration tests
> 3.) anytime we touch UI in this time we must add unit test coverage
> for browser classes (if the page test was acting as a unit test for
> the view)
> 4.) we track closely any regression that slips through and the impact
> of this regression
>
> A note about #2, #3, and #4 here:
>
> For #2, I think we should still run the JS unit tests automatically
> and in my experience Windmill is not fragile for yui tests.  Migrating
> to jstestdriver, if we decide post-experiement to abandon Windmill
> completely, should be a different issue, IMHO.  Francis noted
> yesterday that the API tests for the js LP.client have no other
> coverage, so we should leave those around as well.  Again, I don't
> think these change much or are subject to the same fragility.

So jstestdriver, as it currently stands, is out.  It skips over tests
sometimes, and doesn't even bother to run them.  This is much worse than
windmill's failing of tests that actually pass, in that it could be
missing something important.  Until that is fixed, the jstestdriver talk
should be tabled.

> My assumption about #3 is that we have tested some bits in page tests
> that have no other test coverage.  For example, view methods that
> should be tested in browser unit tests but are covered by the page
> test.  I'm proposing we disable these tests but leave them around and
> that we add unit test coverage where appropriate when touching UI.  I
> am not suggesting we have browser unit tests attempt to replicate
> story tests.  I think it would be could to run the page tests locally
> when working on UI to see if something needs unit test coverage, too.
> But this can't really be enforced by the experiment.

Aaron and I have been doing a pretty good job and testing browser code
in unit tests.  It's not great now, but I have a branch landing that
brings in James Westby's soupmatchers library, so we can accurately test
that things show up on the page properly.

> As for #4, we don't currently track regressions, so I propose that we
> make use of a regression tag going forward.  For *any* regression bug.
>  This will also help with Robert's experiment.  And as we fix these
> bugs we tag them with ui-test-experiment if a page or windmill test
> would have caught this regression.
>
> After the experiment completes, we should make an assessment.  Did the
> decreased test run times affect cycle time?  Did the simpler UI
> testing strategy affect cycle time?  Did we introduce regressions that
> tests would have caught?  Was the impact of such regressions serious
> or minor?  And then, should we continue with this, do something new,
> or re-enable the tests and return to what we had before?
>
> What do you all think?

I think this is a great idea.  I've always thought that windmill tests
are causing more hurt than they are helping (I think putting us in
testfix mode has a bigger cost than I think any graph can demonstrate,
but I've been brainstorming how we can measure it).  The code team has
been staying away from page tests in general, and have been using unit
tests to test similar things.

At UDS, I talked to Curtis about using pagetests as testable
documentation of workflows, but I really couldn't see a need to do so if
we're using unittests to test those.  Unless we have a story for
publishing that documentation (and maybe we want that), I think it's
overhead we don't need to worry about.

Cheers,
Paul