testtools-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxx>]

This email also available at
https://gist.github.com/jml/801103efdf65ee9922fe. Please reply here or
comment there. IRC is tricky for me right now.

Some notes on a different approach to xUnit style tests. I don't
necessarily want to implement it—I don't quite have the motive or
opportunity—but I think it interesting enough to share.

Instead of telling users to subclass ``testtools.TestCase``, we would tell
them to decorate. Tests would look like::

    @test_cases()
    class MyTests(object):
        def setUp(self):
            # no upcall
            whatever()
        def test_foo(self):
            assert 1 == 1
        def tearDown(self):
            destroy_things()

``@test_cases()`` would adapt the suite to an object that implements only
the bare bones of ``TestCase``: ``run()``, ``id()``, ``countTestCases()``,
etc [1]_.

The actual test classes (``MyTests``) would have absolutely _nothing_
inserted into their namespace: no assertions; no fixtures; no details; no
methods to upcall. Maybe we'd have an explicit interface that they would
have to conform to, perhaps implemented by a purely abstract base class.

``@test_cases()`` would only be responsible for the strategy that
translates user code into a runnable ``TestCase``. That is, it would know
about ``setUp`` and ``tearDown``, and it would know how to translate raised
exceptions into various calls on a ``TestResult`` [2]_.

We could do this, we have the technology.

We might even want to go further, and start with something that adapts
*functions* to runnable test cases and then build the richer strategy in
terms of that. That could be quite fun.

If we wanted to provide users with the benefits of details and assertions,
we could build something on top of the basic ``@test_cases`` that passes a
``fixture`` and a ``detailed`` into each method, e.g.::

    @enriched_test_cases(fixtures=True, details=True)
    class MyTests(object):
        def setUp(self, fixture, details):
            fixture.useFixture(TempDir())
        def test_something(self, fixture, details):
            # ...

I think this part needs a lot of experimentation, but I think those
experiments should be done building on top of the minimal one. My hunch is
that it will lead to simpler, more composable code.

We wouldn't need to provide assertions. ``assert_that`` and ``expect_that``
just need to take a ``detailed`` parameter in order to have full
functionality.

Under this regime, ``RunTest`` as interface would melt away. The awkward
constructor interface would be replaced by whatever parameters
``@test_cases`` provides and the ``run()`` interface would no longer be
needed. Much of the code of ``RunTest`` would be moved into the adapter
that ``@test_cases`` uses, but maybe the hierarchy of method calls could be
flattened, since the adapter wouldn't be designed to be inherited from.

Importantly, the relationship would be unidirectional: ``@test_cases``
calls to the actual test case; the test case doesn't know about
``@test_cases`` at all.

``DeferredRunTest`` and friends would likewise become
``@deferred_test_cases`` or something. They wouldn't be obligated to share
code with the synchronous version, but we'd be free to factor out common
code.

Good acceptance tests for this would be whether something like `@flaky
decorator support <
https://github.com/ClusterHQ/flocker/blob/master/flocker/testtools/_flaky.py#L129>`
could be written using only public methods but still work with distinct
underlying runners, whether Hypothesis_ works, whether we could build on
this approach to implement something like testscenarios, and whether we
could move all of the code in ``testtools.TestCase`` out into multiple
smaller components.

Other quick thoughts:

 * We'd have to do some dancing around discovery. I looked briefly at this
and it looks possible.
 * I have no idea how this would work with backwards compatibility. That's
partly why I haven't tried seriously to write this as code.
 * I think this would be easier to do with explicit interfaces
 * Benefits for this plan are mainly implementation-side. End-user benefit
is that they have less magic in their tests and a perhaps a vague feeling
of being more lightweight. Possibly that a compositional approach makes it
easier to build cool stuff for their tests.

Just putting it out there. Thoughts & questions welcome, but please try to
be gentle.

.. [1]
https://github.com/testing-cabal/testtools/pull/178/files#diff-4372c1c31a1c37a51a147d74f52e157fR30
.. [2] In my ideal world, it would translate them to a single value
(perhaps an algebraic data type) that *then* gets translated into calls on
``TestResult``