launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

Hi, all.

I've got a large Tomboy note going for how to write Windmill tests.
It's made of stuff from my lightning talk at the Launchpad
Thunderdome, interleaved with notes made to Tim and Ian recently.  I
paste it here to share knowledge.  I'm also in the process of
reworking this into the Windmill/JavaScript wiki pages.

So then, here lies my notes.  Don't write a Windmill test unless you
embrace the gospel below. ;)


Goal:

We are interested in testing that the XHR for JavaScript on a given
page is hooked up to the server correctly, receives a response, and
that response is generally sane and shown to the user.

That is *all* we test.

Windmill tests are not yet written to be web page smoke tests (i.e. a
replacement for the stuff in stories that exercises the basics of each
page).

• Write the least amount of code you can to achieve this goal
• Think about what the user is doing
• Test what the user will see
• Trust your other tests
• Provide YUI test coverage to exercise the widget itself

We are *not* testing:

• JavaScript
• Widgets
• User interactions

There are other tests to cover these elements.  We only care about the
XHR interactions between web server and the web page.

Some tips to keep you focused on the goal:

1) Avoid waits.sleep

Sleep can't be avoided completely.  Some tests just won't pass without
it.  But you should never use sleep when you can wait on your element
to appear.

If you have to use sleep to get a test to pass, dig deeper and see if
the js or the test can be rewritten.

2) Never use xpath expressions

This is a big one, so some explanation...

== xpath is fragile==

This comes down to developers not understanding how to write xpath
expressions and xpath expressions relying on html.  Since HTML often
changes before id or class names do, this can cause tests to fail and
have to be rewritten, which causes people to trust Windmill tests
less.  Really, this is not so bad on one hand, but it is annoying and
creates work and there are better ways to get a nodes than xpath
anyway.

jquery selectors rock for replacing all the stuff we've used xpath for
in the past.

== xpath is hard to write correctly ==

Worst case, developers write arcane xpath expressions that cause the
test to pass, but they are not really testing what they think they are
testing.  This masks deeper problems with the test.

I've never met someone who *really* knew xpath expressions, have you? ;)

== xpath creates the wrong kind of test ==

It should also be avoided because if you're writing some xpath
expression, you're probably already thinking about testing the wrong
thing.  Windmill tests should be XHR integration tests and they should
be written from the user's point of view.  id and class names help us
identify the parts of the page the user needs to interact with, so all
tests should be written off of these, rather than the html around
these.

In other words, xpath pushes the test writer to think about html
layout and page arrangement, when we really don't care about those in
these tests.  We care about what the user is doing.  Anything the user
clicks or interacts with should have a unique id or can be identified
with a combination of id and class names.  So "avoid xpath" is a good
general rule to producing better tests.  It also produces better code
because you have to think about the id and class names you're using
more, which makes the page, the js, and the widgets themselves better.

== xpath seems slower ==

I don't have benchmarks for this, so I'm hesitant to say it *is*
slower.  But test times have been faster for me, for the times I've
migrated lines away from xpath.  I've never done this in isolation,
i.e. there were other test improvements, too, but my sense is that
xpath takes longer to deal with than selecting on id, classname, or
query selectors.

3) Always use a timeout on waits.forElement

I can't prove it yet, but I think missing timeout values is what
causes Windmill tests to hang.

4) Do not use a page reload to confirm data

Your test can access the same objects as a unit test.  Use that power.
 We want to confirm the XHR post saved the data correctly, so check
the data.  We don't need to confirm via the page.  Just confirm one
small bit on the page, so we know the data was presented back to the
user correctly or that success was accurately reported.

5) Never use assertElemJS or assertProperty

These are not JS or CSS tests remember!  We only care about the server
interactions.  Push this stuff down to YUI tests.

6) Do not do error checking

Use YUI test for making sure the widget does what it should when it errors.



Hope that helps!

Cheers,
deryck