openstack team mailing list archive

[Justin Santa Barbara <justin@xxxxxxxxxxxx>]

I'm really talking about tests that degrade to work around bugs; I agree
that we shouldn't blindly skip whole tests (although when a test is skipped
using @unittest.skip, I think it gives a nice indication that it was skipped
vs passed?)

For example, consider the OpenStack API authentication issue I gave.

*To get a passing unit test with OpenStack authentication, my best bet is to
set all three values (username, api_key, api_secret) to the same value.
 This, however, is a truly terrible test case.  Having "known_bugs" marks my
unit test as being suboptimal; it lets me provide better code in the same
place (controlled by the known_bugs setting); and when the bug fixer comes
to fix it they easily get a failing unit test that they can use for TDD.*

In code, you'd have something like this:

auth_name = random_string(8)
auth_key = random_string(8)
auth_secret = random_string(8)

if 'bug12345' in known_bugs:
  # Set all 3 values the same, so that we can still auth even though secret
& key aren't what we'd intuitively expect
  auth_secret = auth_name
  auth_key = auth_name

create_user(auth_name, auth_key, auth_secret)


So I have a weak test case (because it's the best I can do until bug12345 is
fixed).  When someone removes 'bug12345' from the known_bug list, the same
test case becomes strong.  If someone is ready to fix the OpenStack auth
issue, they need only remove bug12345 from the known_bugs list and the tests
cases will then fail, so they have ready-made TDD.

Another example:  My unit test might skip over an assertion, even though the
spec says it should be the case, because there's a filed bug about it.  I
have the choice between not testing it at all (not writing a good unit
test), or writing a good unit test with a 'guard' that means that it
temporarily degrades to a weaker unit test.

Another example: I encountered an issue with fake_carrot.  Vish fixed it
very quickly, but in the meantime my best bet was to use "real" RabbitMQ.
 Switching between fake_carrot and "real" RabbitMQ could be driven by a
known_bugs flag. This is probably more controversial, but because
fake_carrot is supposed to behave the same as carrot, it did mean I could
satisfy myself & anyone else that the bug was (probably) with fake_carrot,
because it wasn't behaving like the system it was supposed to be faking.


I think in reality there are two alternatives to putting this into a really
obvious file like known_bugs.py:
1) Hide a similar flag in the file with the unit test, or comment out the
'good test code'
or
2) Simply not write a thorough test case.

I think both of these alternatives are worse.  I don't love known_bugs.py
either, but I think it's the best we've got.  If anyone has a better idea,
I'd love to hear it!

Ewan: I think the idea of skipUnless(datetime.now()...) is brilliant.  I'm
just thinking of the havoc that could be caused by sneaking an obfuscated
version of that into the unit test suite, particularly one that only failed
if the username was 'hudson' :)

Justin


On Mon, Feb 28, 2011 at 1:57 PM, Tim Simpson <tim.simpson@xxxxxxxxxxxxx>wrote:

> The Skip plugin for nose offers similar functionality which can be used in
> Python 2.6:
> http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/skip.html
>
> Using this you can write decorators that raise SkipTest if a certain
> criteria isn't met.
>
> ________________________________________
> From: openstack-bounces+tim.simpson=rackspace.com@xxxxxxxxxxxxxxxxxxx[openstack-bounces+tim.simpson=
> rackspace.com@xxxxxxxxxxxxxxxxxxx] on behalf of Clay Gerrard [
> clay.gerrard@xxxxxxxxxxxxx]
> Sent: Monday, February 28, 2011 3:09 PM
> To: Dan Prince; Justin Santa Barbara
> Cc: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Openstack] How to deal with 'tangential' bugs?
>
> Unittest2 lets you define a test case that is expected to fail:
> http://docs.python.org/library/unittest.html#unittest.expectedFailure
>
> new in 2.7, but it could be possible to backport - or do something
> similar...
>
> May have issues with nose:
> http://code.google.com/p/python-nose/issues/detail?id=325
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>