launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Tue, Nov 9, 2010 at 3:38 PM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> This is obviously related to the discussion about how much value we
> get from the tests, but I've not seen any data saying that *windmill*
> is our problem. It seems to be hard to use correctly, and we seem to
> be failing to use it correctly in at least some places. Which sucks
> :(.
>

This may be true for the recent failings, and maybe it's generally
true.  But I don't think this is the whole story.

Tests can be correct and pass by themselves but fail when run as a
group. There have been efforts to rewrite tests and restructure html
to not rely on xpath as well as some other tricks (waits.forElement
after page load rather than assertNode, for example).  So you could
say this is not using Windmill correctly, but "correct" is a sliding
term IMHO.  In other words, we should be using it correctly as we
understand "correct" today, but there's no guarantee this will be
correct tomorrow.

Windmill is notorious about varying results under system load or other
conditions (the 512k file size limitation in the past comes to mind).
Some people cannot run them locally well without really increasing the
timeout constants used.

I have an issue now with getting a lazr-js upgrade branch landed for
similar reasons.  All tests pass individually, but when run as a group
in ec2, 10-15 of them fail at any time. I've been working on this for
a week or better myself, rockstar for two weeks prior to that.  And
yeah, we've made several little changes all along that get us closer
to landing every time. Eventually I'll learn that if I dance on one
leg and turn around in a circle 3 times while throwing salt over my
shoulder and also sacrificing a goat, then the tests will eventually
pass consistently.  But I'm not sure I feel comfortable calling that
incantation the "correct" use of Windmill.

Cheers,
deryck


-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/