testtools-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxx>]

Jonathan Lange has proposed merging lp:~jml/testtools/run-test-improvements into lp:testtools.

Requested reviews:
  testtools developers (testtools-dev)
Related bugs:
  #657760 runTest argument to TestCase unusable
  https://bugs.launchpad.net/bugs/657760
  #657780 Per-test decorator syntax for test runner
  https://bugs.launchpad.net/bugs/657780


This branch makes some changes to how 'RunTest' objects are specified for tests. 

There's not much else to say that the documentation doesn't.  I put the tests in test_runtest, but they could as well go in test_testtools.
-- 
https://code.launchpad.net/~jml/testtools/run-test-improvements/+merge/38659
Your team testtools developers is requested to review the proposed merge of lp:~jml/testtools/run-test-improvements into lp:testtools.

=== modified file 'MANUAL'
--- MANUAL	2010-09-18 02:10:58 +0000
+++ MANUAL	2010-10-17 16:22:40 +0000
@@ -11,11 +11,12 @@
 Extensions to TestCase
 ----------------------
 
-Controlling test execution
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Custom exception handling
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Testtools supports two ways to control how tests are executed. The simplest
-is to add a new exception to self.exception_handlers::
+testtools provides a way to control how test exceptions are handled.  To do
+this, add a new exception to self.exception_handlers on a TestCase.  For
+example::
 
     >>> self.exception_handlers.insert(-1, (ExceptionClass, handler)).
 
@@ -23,12 +24,36 @@
 ExceptionClass, handler will be called with the test case, test result and the
 raised exception.
 
-Secondly, by overriding __init__ to pass in runTest=RunTestFactory the whole
-execution of the test can be altered. The default is testtools.runtest.RunTest
-and calls  case._run_setup, case._run_test_method and finally
-case._run_teardown. Other methods to control what RunTest is used may be
-added in future.
-
+Controlling test execution
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to control more than just how exceptions are raised, you can
+provide a custom `RunTest` to a TestCase.  The `RunTest` object can change
+everything about how the test executes.
+
+To work with `testtools.TestCase`, a `RunTest` must have a factory that takes
+a test and an optional list of exception handlers.  Instances returned by the
+factory must have a `run()` method that takes an optional `TestResult` object.
+
+The default is `testtools.runtest.RunTest` and calls 'setUp', the test method
+and 'tearDown' in the normal, vanilla way that Python's standard unittest
+does.
+
+To specify a `RunTest` for all the tests in a `TestCase` class, do something
+like this::
+
+  class SomeTests(TestCase):
+      run_tests_with = CustomRunTestFactory
+
+To specify a `RunTest` for a specific test in a `TestCase` class, do::
+
+  class SomeTests(TestCase):
+      @run_test_with(CustomRunTestFactory, extra_arg=42, foo='whatever')
+      def test_something(self):
+          pass
+
+In addition, either of these can be overridden by passing a factory in to the
+`TestCase` constructor with the optional 'runTest' argument.
 
 TestCase.addCleanup
 ~~~~~~~~~~~~~~~~~~~

=== modified file 'NEWS'
--- NEWS	2010-10-17 12:54:44 +0000
+++ NEWS	2010-10-17 16:22:40 +0000
@@ -4,6 +4,12 @@
 NEXT
 ~~~~
 
+* Provide a per-test decoractor, run_test_with, to specify which RunTest
+  object to use for a given test.  (Jonathan Lange, #657780)
+
+* Fix the runTest parameter of TestCase to actually work, rather than raising
+  a TypeError.  (Jonathan Lange, #657760)
+
 
 0.9.7
 ~~~~~

=== modified file 'testtools/__init__.py'
--- testtools/__init__.py	2010-10-17 12:55:59 +0000
+++ testtools/__init__.py	2010-10-17 16:22:40 +0000
@@ -11,6 +11,7 @@
     'MultipleExceptions',
     'MultiTestResult',
     'PlaceHolder',
+    'run_test_with',
     'TestCase',
     'TestResult',
     'TextTestResult',
@@ -33,6 +34,7 @@
     PlaceHolder,
     TestCase,
     clone_test_with_new_id,
+    run_test_with,
     skip,
     skipIf,
     skipUnless,

=== modified file 'testtools/testcase.py'
--- testtools/testcase.py	2010-10-14 22:58:49 +0000
+++ testtools/testcase.py	2010-10-17 16:22:40 +0000
@@ -6,10 +6,11 @@
 __all__ = [
     'clone_test_with_new_id',
     'MultipleExceptions',
-    'TestCase',
+    'run_test_with',
     'skip',
     'skipIf',
     'skipUnless',
+    'TestCase',
     ]
 
 import copy
@@ -61,6 +62,29 @@
         """
 
 
+def run_test_with(test_runner, **kwargs):
+    """Decorate a test as using a specific `RunTest`.
+
+    e.g.
+      @run_test_with(CustomRunner, timeout=42)
+      def test_foo(self):
+          self.assertTrue(True)
+
+    :param test_runner: A `RunTest` factory that takes a test case and an
+        optional list of exception handlers.  See `RunTest`.
+    :param **kwargs: Keyword arguments to pass on as extra arguments to
+        `test_runner`.
+    :return: A decorator to be used for marking a test as needing a special
+        runner.
+    """
+    def make_test_runner(case, handlers=None):
+        return test_runner(case, handlers=handlers, **kwargs)
+    def decorator(f):
+        f._run_test_with = make_test_runner
+        return f
+    return decorator
+
+
 class MultipleExceptions(Exception):
     """Represents many exceptions raised from some operation.
 
@@ -74,18 +98,25 @@
     :ivar exception_handlers: Exceptions to catch from setUp, runTest and
         tearDown. This list is able to be modified at any time and consists of
         (exception_class, handler(case, result, exception_value)) pairs.
+    :cvar run_tests_with: A factory to make the `RunTest` to run tests with.
+        Defaults to `RunTest`.  The factory is expected to take a test case
+        and an optional list of exception handlers.
     """
 
     skipException = TestSkipped
 
+    run_tests_with = RunTest
+
     def __init__(self, *args, **kwargs):
         """Construct a TestCase.
 
         :param testMethod: The name of the method to run.
         :param runTest: Optional class to use to execute the test. If not
-            supplied testtools.runtest.RunTest is used. The instance to be
+            supplied `testtools.runtest.RunTest` is used. The instance to be
             used is created when run() is invoked, so will be fresh each time.
+            Overrides `run_tests_with` if given.
         """
+        runTest = kwargs.pop('runTest', None)
         unittest.TestCase.__init__(self, *args, **kwargs)
         self._cleanups = []
         self._unique_id_gen = itertools.count(1)
@@ -95,7 +126,11 @@
         # __details is lazy-initialized so that a constructed-but-not-run
         # TestCase is safe to use with clone_test_with_new_id.
         self.__details = None
-        self.__RunTest = kwargs.get('runTest', RunTest)
+        test_method = self._get_test_method()
+        if runTest is None:
+            runTest = getattr(
+                test_method, '_run_test_with', self.run_tests_with)
+        self.__RunTest = runTest
         self.__exception_handlers = []
         self.exception_handlers = [
             (self.skipException, self._report_skip),
@@ -471,20 +506,22 @@
                 "super(%s, self).tearDown() from your tearDown()."
                 % self.__class__.__name__)
 
-    def _run_test_method(self, result):
-        """Run the test method for this test.
-
-        :param result: A testtools.TestResult to report activity to.
-        :return: None.
-        """
+    def _get_test_method(self):
         absent_attr = object()
         # Python 2.5+
         method_name = getattr(self, '_testMethodName', absent_attr)
         if method_name is absent_attr:
             # Python 2.4
             method_name = getattr(self, '_TestCase__testMethodName')
-        testMethod = getattr(self, method_name)
-        testMethod()
+        return getattr(self, method_name)
+
+    def _run_test_method(self, result):
+        """Run the test method for this test.
+
+        :param result: A testtools.TestResult to report activity to.
+        :return: None.
+        """
+        self._get_test_method()()
 
     def setUp(self):
         unittest.TestCase.setUp(self)

=== modified file 'testtools/tests/test_runtest.py'
--- testtools/tests/test_runtest.py	2010-05-13 12:15:12 +0000
+++ testtools/tests/test_runtest.py	2010-10-17 16:22:40 +0000
@@ -4,10 +4,12 @@
 
 from testtools import (
     ExtendedToOriginalDecorator,
+    run_test_with,
     RunTest,
     TestCase,
     TestResult,
     )
+from testtools.matchers import Is
 from testtools.tests.helpers import ExtendedTestResult
 
 
@@ -176,6 +178,99 @@
             ], result._events)
 
 
+class CustomRunTest(RunTest):
+
+    marker = object()
+
+    def run(self, result=None):
+        return self.marker
+
+
+class TestTestCaseSupportForRunTest(TestCase):
+
+    def test_pass_custom_run_test(self):
+        class SomeCase(TestCase):
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=CustomRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_default_is_runTest_class_variable(self):
+        class SomeCase(TestCase):
+            run_tests_with = CustomRunTest
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_constructor_argument_overrides_class_variable(self):
+        # If a 'runTest' argument is passed to the test's constructor, that
+        # overrides the class variable.
+        marker = object()
+        class DifferentRunTest(RunTest):
+            def run(self, result=None):
+                return marker
+        class SomeCase(TestCase):
+            run_tests_with = CustomRunTest
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=DifferentRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+    def test_decorator_for_run_test(self):
+        # Individual test methods can be marked as needing a special runner.
+        class SomeCase(TestCase):
+            @run_test_with(CustomRunTest)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_extended_decorator_for_run_test(self):
+        # Individual test methods can be marked as needing a special runner.
+        # Extra arguments can be passed to the decorator which will then be
+        # passed on to the RunTest object.
+        marker = object()
+        class FooRunTest(RunTest):
+            def __init__(self, case, handlers=None, bar=None):
+                super(FooRunTest, self).__init__(case, handlers)
+                self.bar = bar
+            def run(self, result=None):
+                return self.bar
+        class SomeCase(TestCase):
+            @run_test_with(FooRunTest, bar=marker)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+    def test_constructor_overrides_decorator(self):
+        # If a 'runTest' argument is passed to the test's constructor, that
+        # overrides the decorator.
+        marker = object()
+        class DifferentRunTest(RunTest):
+            def run(self, result=None):
+                return marker
+        class SomeCase(TestCase):
+            @run_test_with(CustomRunTest)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=DifferentRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+
 def test_suite():
     from unittest import TestLoader
     return TestLoader().loadTestsFromName(__name__)

=== modified file 'testtools/tests/test_testresult.py'
--- testtools/tests/test_testresult.py	2010-10-14 22:50:38 +0000
+++ testtools/tests/test_testresult.py	2010-10-17 16:22:40 +0000
@@ -406,7 +406,7 @@
   File "...testtools...runtest.py", line ..., in _run_user...
     return fn(*args)
   File "...testtools...testcase.py", line ..., in _run_test_method
-    testMethod()
+    self._get_test_method()()
   File "...testtools...tests...test_testresult.py", line ..., in error
     1/0
 ZeroDivisionError:... divi... by zero...
@@ -420,7 +420,7 @@
   File "...testtools...runtest.py", line ..., in _run_user...
     return fn(*args)
   File "...testtools...testcase.py", line ..., in _run_test_method
-    testMethod()
+    self._get_test_method()()
   File "...testtools...tests...test_testresult.py", line ..., in failed
     self.fail("yo!")
 AssertionError: yo!