launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

On Tue, Jun 14, 2011 at 6:07 PM, Curtis Hovey
<curtis.hovey@xxxxxxxxxxxxx> wrote:
> YUI UnitTesting is reintegrated in Launchpad's test suite. There are
> substantial changes that you need to know when working with JavaScript.
>
> Note: We are waiting on rt #46284 to be completed. When pqm and buildbot
> have the new test runner image, I will land a branch to enable YUI
> unittest to be run by default.
>
> Launchpad developer dependencies were updated a few days ago to require
> html5browser. This library allows us to drive a modern browser from
> python. It is based on webkitgtk. This is a html5/css3/es3 compatible
> engine that is used by Chromium and Safari. You will not see a browser
> while the test runs, but since this is a gtk library, xvfb is needed by
> headless machines.
>
> You can run all YUI unittests with
>    ./bin/test -vvc --layer=YUI
> The tests will complete in about 45 seconds.
>
> The YUITestLayer extends FunctionalLayer. It requires that the tree is
> built. It does not require an app server, Firefox, or windmill. The base
> testcase does not poll or wait for the test to complete. The browser
> connects to a the status-changed event which is sent by the YUI unittest
> on complete. Assigning string that starts with '::::' will return the
> string to the Python test.:
>    var handle_complete = function(data) {
>        window.status = '::::' + JSON.stringify(data);
>        };
>    Y.Test.Runner.on('complete', handle_complete);
> See https://dev.launchpad.net/JavascriptUnitTesting for the full
> example. Setting the window.status to a string without '::::' will not
> affect the Python test.
>
> The Python testcase expects a JSON encoded results. I think the approach
> is fast and robust. Previously it waited for a DOM change then attempted
> to parse the page to get the log. I discovered that YUI unittest will
> fire the complete event *before* the log is rendered. This explains why
> so many YUI tests fail on Jenkins, but do not *appear* to fail when we
> view them in out browser. YUI tests that simulated events we often
> affected by they issue.
>
> I was surprised that many YUI unittests do not complete in under 5
> seconds. There are two causes for this. Many test modules are composed
> of many testcases; structural subscription tests for example could be
> broken into several modules that complete quickly. I see a few tests
> that are calling wait(). This looks scary. I do not know the reason for
> this since the tests do not use a web server. Maybe the tests should
> listen for a signal instead. I set the base Python unittest case to
> increase the default browser timeout from 5 seconds to 30. This change
> does not slow down the test runner. You may discover your test additions
> cause a timeout failures.
>

Curtis, thanks so much for doing this work!  I am very excited to see this land.

As for the question of waits in unit tests, I believe people are doing
some DOM animation/redrawing and want to be sure the work is completed
before asserting.  I think this makes sense on one hand, but we should
make sure we absolutely need it before using it.  e.g. if you're
asserting a class that changes immediately, even while the animation
runs, you don't have to wait on the animation to complete to do the
assert.  Also, we should have a hard requirement that waits in js unit
tests, if required, should be on the order of 5-10 ms, not in the 100s
of ms.  If we need to wait that long, we should fix the widget.  The
user experience will be slow as well as the test if the widget is that
slow changing the page.

Again, thanks for making this possible.  It's a great step forward in
unit testing our JavaScript!

Cheers,
deryck




-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/