testtools-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

Robert Collins has proposed merging lp:~lifeless/testtools/preserve-suites-load-list into lp:testtools.

Requested reviews:
  testtools committers (testtools-committers)

For more details, see:
https://code.launchpad.net/~lifeless/testtools/preserve-suites-load-list/+merge/144011

Foinally.
-- 
https://code.launchpad.net/~lifeless/testtools/preserve-suites-load-list/+merge/144011
Your team testtools developers is subscribed to branch lp:testtools.

=== modified file 'NEWS'
--- NEWS	2013-01-18 11:24:57 +0000
+++ NEWS	2013-01-20 07:34:21 +0000
@@ -9,6 +9,11 @@
 Changes
 -------
 
+* ``python -m testtools.run --load-list`` will now preserve any custom suites
+  (such as ``testtools.FixtureSuite`` or ``testresources.OptimisingTestSuite``)
+  rather than flattening them.
+  (Robert Collins, #827175)
+
 * Testtools now depends on extras, a small library split out from it to contain
   generally useful non-testing facilities. Since extras has been around for a
   couple of testtools releases now, we're making this into a hard dependency of

=== modified file 'doc/for-framework-folk.rst'
--- doc/for-framework-folk.rst	2012-12-18 09:03:19 +0000
+++ doc/for-framework-folk.rst	2013-01-20 07:34:21 +0000
@@ -21,6 +21,10 @@
 Extensions to TestCase
 ======================
 
+In addition to the ``TestCase`` specific methods, we have extensions for
+``TestSuite`` that also apply to ``TestCase`` (because ``TestCase`` and
+``TestSuite`` follow the Composite pattern).
+
 Custom exception handling
 -------------------------
 
@@ -220,7 +224,10 @@
 
 A test suite that sets up a fixture_ before running any tests, and then tears
 it down after all of the tests are run. The fixture is *not* made available to
-any of the tests.
+any of the tests due to there being no standard channel for suites to pass
+information to the tests they contain (and we don't have enough data on what
+such a channel would need to achieve to design a good one yet - or even decide
+if it is a good idea).
 
 sorted_tests
 ------------
@@ -229,9 +236,21 @@
 problematic - you can't tell what functionality is embedded into custom Suite
 implementations. In order to deliver consistent test orders when using test
 discovery (see http://bugs.python.org/issue16709), testtools flattens and
-sorts tests that have the standard TestSuite, defines a new method sort_tests,
-which can be used by non-standard TestSuites to know when they should sort
-their tests.
+sorts tests that have the standard TestSuite, and defines a new method
+sort_tests, which can be used by non-standard TestSuites to know when they
+should sort their tests. An example implementation can be seen at
+``FixtureSuite.sorted_tests``.
+
+filter_by_ids
+-------------
+
+Simiarly to ``sorted_tests`` running a subset of tests is problematic - the
+standard run interface provides no way to limit what runs. Rather than
+confounding the two problems (selection and execution) we defined a method
+that filters the tests in a suite (or a case) by their unique test id.
+If you a writing custom wrapping suites, consider implementing filter_by_ids
+to support this (though most wrappers that subclass ``unittest.TestSuite`` will
+work just fine [see ``testtools.testsuite.filter_by_ids`` for details.]
 
 .. _`testtools API docs`: http://mumak.net/testtools/apidocs/
 .. _unittest: http://docs.python.org/library/unittest.html

=== modified file 'testtools/run.py'
--- testtools/run.py	2012-12-18 09:03:19 +0000
+++ testtools/run.py	2013-01-20 07:34:21 +0000
@@ -14,7 +14,7 @@
 
 from testtools import TextTestResult
 from testtools.compat import classtypes, istext, unicode_output_stream
-from testtools.testsuite import iterate_tests, sorted_tests
+from testtools.testsuite import filter_by_ids, iterate_tests, sorted_tests
 
 
 defaultTestLoader = unittest.defaultTestLoader
@@ -173,11 +173,7 @@
             finally:
                 source.close()
             test_ids = set(line.strip().decode('utf-8') for line in lines)
-            filtered = unittest.TestSuite()
-            for test in iterate_tests(self.test):
-                if test.id() in test_ids:
-                    filtered.addTest(test)
-            self.test = filtered
+            self.test = filter_by_ids(self.test, test_ids)
         if not self.listtests:
             self.runTests()
         else:

=== modified file 'testtools/tests/test_run.py'
--- testtools/tests/test_run.py	2013-01-18 09:17:19 +0000
+++ testtools/tests/test_run.py	2013-01-20 07:34:21 +0000
@@ -5,15 +5,15 @@
 from unittest import TestSuite
 
 from extras import try_import
+fixtures = try_import('fixtures')
+testresources = try_import('testresources')
 
+import testtools
+from testtools import TestCase, run
 from testtools.compat import (
     _b,
     StringIO,
     )
-fixtures = try_import('fixtures')
-
-import testtools
-from testtools import TestCase, run
 from testtools.matchers import Contains
 
 
@@ -43,6 +43,50 @@
             self.addCleanup(testtools.__path__.remove, self.package.base)
 
 
+if fixtures and testresources:
+    class SampleResourcedFixture(fixtures.Fixture):
+        """Creates a test suite that uses testresources."""
+
+        def __init__(self):
+            super(SampleResourcedFixture, self).__init__()
+            self.package = fixtures.PythonPackage(
+            'resourceexample', [('__init__.py', _b("""
+from fixtures import Fixture
+from testresources import (
+    FixtureResource,
+    OptimisingTestSuite,
+    ResourcedTestCase,
+    )
+from testtools import TestCase
+
+class Printer(Fixture):
+
+    def setUp(self):
+        super(Printer, self).setUp()
+        print('Setting up Printer')
+
+class TestFoo(TestCase, ResourcedTestCase):
+    # When run, this will print just one Setting up Printer, unless the
+    # OptimisingTestSuite is not honoured, when one per test case will print.
+    resources=[('res', FixtureResource(Printer()))]
+    def test_bar(self):
+        pass
+    def test_foo(self):
+        pass
+    def test_quux(self):
+        pass
+def test_suite():
+    from unittest import TestLoader
+    return OptimisingTestSuite(TestLoader().loadTestsFromName(__name__))
+"""))])
+
+        def setUp(self):
+            super(SampleResourcedFixture, self).setUp()
+            self.useFixture(self.package)
+            self.addCleanup(testtools.__path__.remove, self.package.base)
+            testtools.__path__.append(self.package.base)
+
+
 class TestRun(TestCase):
 
     def setUp(self):
@@ -100,6 +144,33 @@
         self.assertEqual("""testtools.runexample.TestFoo.test_bar
 """, out.getvalue())
 
+    def test_load_list_preserves_custom_suites(self):
+        if testresources is None:
+            self.skipTest("Need testresources")
+        self.useFixture(SampleResourcedFixture())
+        # We load two tests, not loading one. Both share a resource, so we
+        # should see just one resource setup occur.
+        tempdir = self.useFixture(fixtures.TempDir())
+        tempname = tempdir.path + '/tests.list'
+        f = open(tempname, 'wb')
+        try:
+            f.write(_b("""
+testtools.resourceexample.TestFoo.test_bar
+testtools.resourceexample.TestFoo.test_foo
+"""))
+        finally:
+            f.close()
+        stdout = self.useFixture(fixtures.StringStream('stdout'))
+        with fixtures.MonkeyPatch('sys.stdout', stdout.stream):
+            try:
+                run.main(['prog', '--load-list', tempname,
+                    'testtools.resourceexample.test_suite'], stdout.stream)
+            except SystemExit:
+                # Evil resides in TestProgram.
+                pass
+        out = stdout.getDetails()['stdout'].as_text()
+        self.assertEqual(1, out.count('Setting up Printer'), "%r" % out)
+
     def test_run_failfast(self):
         stdout = self.useFixture(fixtures.StringStream('stdout'))
 

=== modified file 'testtools/testsuite.py'
--- testtools/testsuite.py	2013-01-18 09:17:19 +0000
+++ testtools/testsuite.py	2013-01-20 07:34:21 +0000
@@ -5,6 +5,7 @@
 __metaclass__ = type
 __all__ = [
   'ConcurrentTestSuite',
+  'filter_by_ids',
   'iterate_tests',
   'sorted_tests',
   ]
@@ -147,6 +148,66 @@
         return [(suite_id, suite_or_case)]
 
 
+def filter_by_ids(suite_or_case, test_ids):
+    """Remove tests from suite_or_case where their id is not in test_ids.
+    
+    :param suite_or_case: A test suite or test case.
+    :param test_ids: Something that supports the __contains__ protocol.
+    :return: suite_or_case, unless suite_or_case was a case that itself
+        fails the predicate when it will return a new unittest.TestSuite with
+        no contents.
+
+    This helper exists to provide backwards compatability with older versions
+    of Python (currently all versions :)) that don't have a native
+    filter_by_ids() method on Test(Case|Suite).
+
+    For subclasses of TestSuite, filtering is done by:
+        - attempting to call suite.filter_by_ids(test_ids)
+        - if there is no method, iterating the suite and identifying tests to
+          remove, then removing them from _tests, manually recursing into
+          each entry.
+
+    For objects with an id() method - TestCases, filtering is done by:
+        - attempting to return case.filter_by_ids(test_ids)
+        - if there is no such method, checking for case.id() in test_ids
+          and returning case if it is, or TestSuite() if it is not.
+
+    For anything else, it is not filtered - it is returned as-is.
+
+    To provide compatability with this routine for a custom TestSuite, just
+    define a filter_by_ids() method that will return a TestSuite equivalent to
+    the original minus any tests not in test_ids.
+    Similarly to provide compatability for a custom TestCase that does
+    something unusual define filter_by_ids to return a new TestCase object
+    that will only run test_ids that are in the provided container. If none
+    would run, return an empty TestSuite().
+
+    The contract for this function does not require mutation - each filtered
+    object can choose to return a new object with the filtered tests. However
+    because existing custom TestSuite classes in the wild do not have this
+    method, we need a way to copy their state correctly which is tricky:
+    thus the backwards-compatible code paths attempt to mutate in place rather
+    than guessing how to reconstruct a new suite.
+    """
+    # Compatible objects
+    if safe_hasattr(suite_or_case, 'filter_by_ids'):
+        return suite_or_case.filter_by_ids(test_ids)
+    # TestCase objects.
+    if safe_hasattr(suite_or_case, 'id'):
+        if suite_or_case.id() in test_ids:
+            return suite_or_case
+        else:
+            return unittest.TestSuite()
+    # Standard TestSuites or derived classes [assumed to be mutable].
+    if isinstance(suite_or_case, unittest.TestSuite):
+        filtered = []
+        for item in suite_or_case:
+            filtered.append(filter_by_ids(item, test_ids))
+        suite_or_case._tests[:] = filtered
+    # Everything else:
+    return suite_or_case
+
+
 def sorted_tests(suite_or_case, unpack_outer=False):
     """Sort suite_or_case while preserving non-vanilla TestSuites."""
     tests = _flatten_tests(suite_or_case, unpack_outer=unpack_outer)