testtools-dev team mailing list archive

[Daniel Watkins <daniel@xxxxxxxxxxxxxxxxxxxx>]

Hello all,

As you are probably aware, the assertions in unittest.TestCase will
output a diff for certain types of comparison.  As I understand it,
testtools currently doesn't do this under any circumstances.

This is a Bad Thing (TM) for two reasons: firstly, it makes it harder to
identify minor differences between expected and actual output; this is a
usability issue.  Secondly, as the Python stdlib already supports it, it
is probably something that testtools will need to support to be a
serious candidate for inclusion in the stdlib.

The actual output of the diff is, I think, relatively straightforward;
the Mismatches concerned would need some changes in their describe
method (and some new, more specific Mismatches might be needed).  I
assume that pull requests to implement this would be welcome (please
correct me if not!).

The difficult part would (I think) come when trying to support the
existing maxDiff property that unittest.TestCase supports[0].  This
defines the largest diff that should be output (with a "this diff is too
long" message being output if it is exceeded), and part of the public
interface of the TestCase class.  Personally, I don't think that the
TestCase should be responsible for the presentation of assertion output
(and therefore I dislike maxDiff being an instance attribute).  It seems
that there's a level of agreement from the testtools developers, as
assertThat takes a verbose argument (rather than relying on something on
the TestCase instance)[1].

I would be loath, however, to just add a max diff keyword argument to
assertThat; that way madness lies[2].  I therefore propose changing the
public API of assertThat to take a format_options argument which would
describe all of the formatting details that the user desires.  I would
suggest that this be a new class (called, perhaps, AssertionFormat)
which can handle defaults sensibly (making changes/additions to the
formatting interface less painful).

In order to support the existing unittest.TestCase assertion interface,
we would construct an AssertionFormat within the assertion and pass that
through to self.assertThat:

    def assertEqual(self, expected, observed, message=''):
        matcher = Equals(expected)
        format_options = FormatOptions(max_diff=self.maxDiff)
        self.assertThat(observed,
                        matcher,
                        message,
                        format_options=format_options)


assertThat remains insulated from the maxDiff nonsense, and it can go
the way of the dodo if/when the bulk of assert* methods on TestCase
disappear.  We would do something similar with the current verbose
argument:

    def assertThat(self, matchee, matcher, message='', verbose=False, format_options=None):
        ...
        if format_options is None:
            format_options = FormatOptions(verbose=verbose)
        raise MismatchError(matchee, matcher, mismatch, format_options)

Obviously there are some rough edges to work out here; this is just to
give you an idea of the rough drift of my proposal.

If people want to maintain the existing "all of my assertions share a
set of formatting options" behaviour, then they can subclass
testtools.TestCase, and override assertThat to pass their format object
to the super-class's implementation; we could potentially provide a
FormattingTestCase which does this using instance attributes (though I'm
not sure I would bother; happy to be convinced otherwise).


Thoughts/comments/criticisms/abuse^Wcompliments welcome!


Cheers,

Dan


[Footnote 0: There's also _diffThreshold, which defines a bound on the
             largest strings that unittest.TestCase should try to
             compute diffs for, to avoid spending too long on diff
             generation.  This is private, though, so I don't think we
             need to maintain it.]

[Footnote 1: I'm also somewhat biased against storing the formatting
             options on the class because it means that people who want
             to use assertions as a function are limited to a sub-set of
             functionality.]

[Footnote 2: What happens when we need to never display a diff under a
             _minimum_ size?  Or when we want our output to
             include/exclude colour codes?  Or some other thing that I
             haven't thought of?]