openstack team mailing list archive

[Ewan Mellor <Ewan.Mellor@xxxxxxxxxxxxx>]

Python 2.7 has @unittest.skip and @unittest.skipUnless decorators.  Is this what you want?  You could write the failing unit test, and then mark it as skipped until the bug is fixed.  My only concern would be the Python 2.7 dependency - we're using 2.6 still ourselves, so I'd ask that you wrote some backwards-compat code for that.

You could even have skipUnless(datetime.now() - datetime(2011,3,1) > timedelta(0)), so if someone promised you that they were going to fix it today, you could hold them to it!  (I don't recommend that, by the way, but I thought that it was fun ;-)

Ewan.

From: openstack-bounces+ewan.mellor=citrix.com@xxxxxxxxxxxxxxxxxxx [mailto:openstack-bounces+ewan.mellor=citrix.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Justin Santa Barbara
Sent: 28 February 2011 10:56
To: openstack@xxxxxxxxxxxxxxxxxxx
Subject: [Openstack] How to deal with 'tangential' bugs?

Jay and I have been having an interesting discussion about how to deal with bugs that mean that unit tests _should_ fail.  So, if I find a bug, I should write a failing unit test first, and fix it (in one merge).  However, if I can't fix it, I can't get a failing unit test merged into the trunk (because it fails).  It may be that I can't get what I'm actually working on merged with good unit tests until this 'tangential' bug is fixed.

(The discussion is here: https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227)

I suggested that we introduce a "known_bugs" collection.  It would have a set of values to indicate bugs that are known but not yet fixed.  Ideally these would be linked to bug reports (we could mandate this).  When a developer wants to write a test or behavior to work around a particular bug, they can control it based on testing this collection ("if 'bug12345' in known_bugs:")  When someone is ready to fix the bug, they remove the bug from the collection, the unit tests then fail, they fix the code and commit with the known_bugs item removed.

This would let people that find bugs but can't or don't want to fix them still contribute unit tests.  This could be a QA person that can write tests but not necessarily code the fix.  This could be a developer who simply isn't familiar with the particular system.  Or it could be where the fix needs to go through the OpenStack discussion process.  Or it could simply be a train-of-thought / 'flow' issue.

Take, for example, my favorite OpenStack API authentication issue.  To get a passing unit test with OpenStack authentication, my best bet is to set all three values (username, api_key, api_secret) to the same value.  This, however, is a truly terrible test case.  Having "known_bugs" marks my unit test as being suboptimal; it lets me provide better code in the same place (controlled by the known_bugs setting); and when the bug fixer comes to fix it they easily get a failing unit test that they can use for TDD.

Jay (correctly) points out that this is complicated; can cause problems down the line when the bug is fixed in an unexpected way; that known_bugs should always be empty; and that the right thing to do is to fix the bug or get it fixed.  I agree, but I don't think that getting the bug fixed before proceeding is realistic in a project with as many stakeholders as OpenStack has.

Can we resolve the dilemma?  How should we proceed when we find a bug but we're working on something different?

Justin