testtools-dev team mailing list archive

[Robert Collins <654474@xxxxxxxxxxxxxxxxxx>]

Ok, so you want to add another status code to the stream.

unexpectedsuccess is a combination of:
 - behaviour (the test did not raise an assertion)
 - expectation (the test was expected to raise an assertion)

We have the following status codes:
success (no exception, none expected)
failure ('assertion' exception, none expected)
error (other exception, none expected)
skip (not executed, no exception expected)
xfail (any exception, exception expected)

so adding a new status code does round that out.  I remain dissatisfied
with the modelling though; I would kind of like to separate the observed
behaviour and the expected behaviour; a bit like jelmers patch to coerce
xfail to success.

Something like:
status code becomes pure behavior:
completed
raised exception
not executed

expected behaviour becomes:
not supplied
completes
raises exception
not executed

unexpected success then is:
(outcome=completed, expected_outcome=raises exception)
xfail is:
(outcome=raises exception, expected_outcome=raises exception)

One problem with this is that xfail may be quite specific (raises /this/
exception) but subunit doesn't know enough to process that. A subject
for TIP I think.

-- 
You received this bug notification because you are a member of testtools
developers, which is subscribed to testtools.
https://bugs.launchpad.net/bugs/654474

Title:
  An unexpectedSuccess is like a failure not a success

Status in Bazaar Version Control System:
  Confirmed
Status in SubUnit:
  Confirmed
Status in testtools:
  Fix Released

Bug description:
  When the function passed to expectedFailure did not raise an
  assertion, bzr used to do:

      self.fail('Unexpected success.  Should have failed: %s' % reason)

  With testtools, bazaar now treats that as a success, as jam noted in February:
  <https://lists.ubuntu.com/archives/bazaar/2010q1/066898.html>

  And in September he pointed out the same thing, but with subunit:
  <https://lists.ubuntu.com/archives/bazaar/2010q3/070192.html>