launchpad-dev team mailing list archive

[Ian Booth <ian.m.booth@xxxxxxxxx>]

Tim beat me to the keyboard here, so let me add my vote to what he says.
Without Windmill, how do we propose to test stuff like AJAX
functionality which AIUI we must test in an actual browser? I think we
expect the amount of AJAX usage in Launchpad to increase? So taking away
a means to test that would be a problem I would think.

I also agree with other comments that it is less than desirable to
remove tests which do serve a useful purpose. Windmill tests are indeed
integration tests which exercise the entire stack end-end. The value of
this should not be underestimated for two reasons: 1. we do not seem to
have 100% coverage using alternate testing mechanisms and this type of
test is a great safety net since a breakage that may not be caught
elsewhere will often show up here; 2. full end-end system testing is
often the only way to catch certain classes of error which are difficult
to pick up with just unit or module tests. Add to that that as stated
above, there's currently no other alternative for AJAX testing etc.

As others have said, I think it comes down to the notion that Windmill
tests should be removed as a blocker for landing but not rollout. So +1
me on that. My view is:
- Windmill tests should stay for the reasons stated above
- they should be looked at during any code review
- they should pass locally before landing, where the robustness issues
associated with system load will be minimised
- they should be run out-of-band and also act as a blocker to prior to
rollout


On 03/11/10 08:50, Tim Penhey wrote:
> On Wed, 03 Nov 2010 07:18:32 Gary Poster wrote:
>> On Nov 2, 2010, at 1:56 PM, Deryck Hodge wrote:
>>> FWIW, while the pain of Windmill has been acute lately, I don't recall
>>> a month without some form of Windmill pain since I've been working on
>>> Launchpad.
>>
>> Point taken.
>>
>> I'm still not convinced myself, and would much prefer the
>> integration-tests-for-deployment story, but I'm just a vote.
>>
>>> I feel the same about page tests.
>>
>> Something I should have said in my first reply is that discarding the
>> Windmill tests worries me; discarding the page tests frightens me, unless
>> the unit test coverage is much, much higher than I believe it to be.  I
>> think these two parts of your proposal should be separate.
> 
> I have to agree with Gary here.
> 
> While I'm +1 on breaking out the windmill and pagetests to acceptance tests 
> that block rollout but not landing, I can't agree that we should just turn 
> them off completely.
> 
> We, in the code team, have been doing more unit style tests for our browser 
> code, but we in no way have enough test coverage that way for me to be 
> entirely comfortable.
> 
> The page tests often test workflows, which are getting more out of date with 
> the addition of more ajax stuff, but I think it still has some, but limited, 
> use.
> 
> Tim
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~launchpad-dev
> Post to     : launchpad-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~launchpad-dev
> More help   : https://help.launchpad.net/ListHelp
>