openstack team mailing list archive

[Justin Santa Barbara <justin@xxxxxxxxxxxx>]

Jay and I have been having an interesting discussion about how to deal with
bugs that mean that unit tests _should_ fail.  So, if I find a bug, I should
write a failing unit test first, and fix it (in one merge).  However, if I
can't fix it, I can't get a failing unit test merged into the trunk (because
it fails).  It may be that I can't get what I'm actually working on merged
with good unit tests until this 'tangential' bug is fixed.

(The discussion is here:
https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227)

I suggested that we introduce a "known_bugs" collection.  It would have a
set of values to indicate bugs that are known but not yet fixed.  Ideally
these would be linked to bug reports (we could mandate this).  When a
developer wants to write a test or behavior to work around a particular bug,
they can control it based on testing this collection ("if 'bug12345' in
known_bugs:")  When someone is ready to fix the bug, they remove the bug
from the collection, the unit tests then fail, they fix the code and commit
with the known_bugs item removed.

This would let people that find bugs but can't or don't want to fix them
still contribute unit tests.  This could be a QA person that can write tests
but not necessarily code the fix.  This could be a developer who simply
isn't familiar with the particular system.  Or it could be where the fix
needs to go through the OpenStack discussion process.  Or it could simply be
a train-of-thought / 'flow' issue.

Take, for example, my favorite OpenStack API authentication issue.  To get a
passing unit test with OpenStack authentication, my best bet is to set all
three values (username, api_key, api_secret) to the same value.  This,
however, is a truly terrible test case.  Having "known_bugs" marks my unit
test as being suboptimal; it lets me provide better code in the same place
(controlled by the known_bugs setting); and when the bug fixer comes to fix
it they easily get a failing unit test that they can use for TDD.

Jay (correctly) points out that this is complicated; can cause problems down
the line when the bug is fixed in an unexpected way; that known_bugs should
always be empty; and that the right thing to do is to fix the bug or get it
fixed.  I agree, but I don't think that getting the bug fixed before
proceeding is realistic in a project with as many stakeholders as OpenStack
has.

Can we resolve the dilemma?  How should we proceed when we find a bug but
we're working on something different?

Justin