launchpad-dev team mailing list archive

[Edwin Grubbs <edwin.grubbs@xxxxxxxxxxxxx>]

On Mon, Sep 21, 2009 at 5:11 AM, Bjorn Tillenius <bjorn@xxxxxxxxxxxxx> wrote:
> When 3.0 is out of the way, the next priority will be to get the
> currently failing Windmill test suite to pass.
>
> I have made some changes to the way Windmill tests are run, but old
> tests need to be converted in order to have them run in the new way. So
> the way you will make sure that a Windmill test passes will be:
>
>    1. Convert the test to a TestCase test
>    2. Run 'make jscheck'
>
> 'make jscheck' runs only the new-style Windmill tests, and that test
> suite currently passes. There's a buildbot running 'make jscheck'
> daily:
>
>    https://lpbuildbot.canonical.com/builders/jscheck
>
> The old Windmill tests are run using the functest test runner. We want
> to get rid of that test runner, and instead use our own bin/test test
> runner. In order to do this, we need to convert the old tests to be
> layered TestCase tests.
>
> You can see examples of a new-style Windmill tests in
> lib/lp/bugs/windmill/tests/test_bug_commenting.py and
> lib/lp/code/windmill/tests/test_branch_links.py
>
> The process of converting the tests is basically:
>
>    1. Make sure that YourApplicationWindmillLayer exists.
>       If not, create one in lp.yourapplication.windmill.testing. See
>       lp.bugs.windmill.testing.py for an example.
>
>    2. Make sure that bin/test --layer=YourApplicationWindmillLayer is
>       run by 'make jscheck' (Check in Makefile)
>
>    3. Move the python file to lp.yourapplication.windmill.tests. The
>       test_foo.py file has to be directly under a tests/ directory,
>       otherwise it won't be picked up.
>
>    4. Create a subclass of TestCaseWithFactory, setting
>       layer = YourApplicationWindmillLayer.
>
>    5. Create a test_foo() method, moving the contents of the old
>       Windmill test function inside it.
>
> And that's it really. You can now run the test using bin/test
> --layer=YourApplicationWindmillLayer -t test_foo. Specifying the layer
> is important, since by default bin/test won't run tests in a
> WindmillLayer.
>
> As a bonus, you can now use the LaunchpadObjectFactory to do test setup.
> Don't forget to commit transaction after the setup, since otherwise the
> app server won't see the changes.
>
> The db also gets reset between tests, so you don't have to clean up
> after yourselves in the test.
>
> Please let me know if you run into any problems.
>
> Known issues:
>
>    * If don't create a new branch, you might have to re-generate
>      bin/test, since there were changes in test.in, and bin/test
>      doesn't get re-genereated automatically. Removing bin/buildout and
>      running 'make' fixes it. The symptom here is that bin/test
>      --layer=WindmillLayer will run all the tests.
>
>    * Running 'bin/test --layer=WindmillLayer' or running two tests in
>      different WindmillLayer Hangs when the second layer is set up.
>
>    * Sometimes an exception is raised (complaining about invalid thread
>      ids) during teardown.
>
> I have submitted fixes for the two latter issues upstream, and hopefully
> it will be included in Windmill 1.3, so it will be fixes when we
> upgrade. If it causes too much of a problem, I'll look into patching our
> version locally.


I am in the process of converting the registry windmill tests to use
./bin/test, and I am having problems with sporadic
LayerIsolationErrors. I put sleep() calls in the tearDown(), which
reduced the frequency of errors from 10 out of 20 to 1 or 2 out of 20.
I'm already using a 9 second sleep, so I think there must be a better
way to prevent the errors.

Error in test lp.registry.windmill.tests.test_timeline_graph.TestTimelineGraph.test_series_timeline_graph
Traceback (most recent call last):
  File "/home/egrubbs/canonical/lp-branches/bug-430708-registry-windmill-layer/lib/canonical/testing/layers.py",
line 394, in flagTestIsolationFailure
    raise LayerIsolationError(message)
LayerIsolationError: Test left new live threads: [<Thread(Thread-285,
started daemon)>]


The above error occurs most of the time after running
test_series_timeline_graph, but if I run that test by itself, I never
get any LayerIsolationErrors. You can see my changes in this branch:

lp:~edwin-grubbs/launchpad/bug-430708-registry-windmill-layer

I had to add some sleep() calls inside a couple of the windmill tests
to prevent sporadic failures where it can't find a node. This is odd,
since I put a waits.forElement() call directly before the
assertNode(), so it should never fail to find a node.

-Edwin