openstack team mailing list archive
Mailing list archive
How to deal with 'tangential' bugs?
Jay and I have been having an interesting discussion about how to deal with
bugs that mean that unit tests _should_ fail. So, if I find a bug, I should
write a failing unit test first, and fix it (in one merge). However, if I
can't fix it, I can't get a failing unit test merged into the trunk (because
it fails). It may be that I can't get what I'm actually working on merged
with good unit tests until this 'tangential' bug is fixed.
(The discussion is here:
I suggested that we introduce a "known_bugs" collection. It would have a
set of values to indicate bugs that are known but not yet fixed. Ideally
these would be linked to bug reports (we could mandate this). When a
developer wants to write a test or behavior to work around a particular bug,
they can control it based on testing this collection ("if 'bug12345' in
known_bugs:") When someone is ready to fix the bug, they remove the bug
from the collection, the unit tests then fail, they fix the code and commit
with the known_bugs item removed.
This would let people that find bugs but can't or don't want to fix them
still contribute unit tests. This could be a QA person that can write tests
but not necessarily code the fix. This could be a developer who simply
isn't familiar with the particular system. Or it could be where the fix
needs to go through the OpenStack discussion process. Or it could simply be
a train-of-thought / 'flow' issue.
Take, for example, my favorite OpenStack API authentication issue. To get a
passing unit test with OpenStack authentication, my best bet is to set all
three values (username, api_key, api_secret) to the same value. This,
however, is a truly terrible test case. Having "known_bugs" marks my unit
test as being suboptimal; it lets me provide better code in the same place
(controlled by the known_bugs setting); and when the bug fixer comes to fix
it they easily get a failing unit test that they can use for TDD.
Jay (correctly) points out that this is complicated; can cause problems down
the line when the bug is fixed in an unexpected way; that known_bugs should
always be empty; and that the right thing to do is to fix the bug or get it
fixed. I agree, but I don't think that getting the bug fixed before
proceeding is realistic in a project with as many stakeholders as OpenStack
Can we resolve the dilemma? How should we proceed when we find a bug but
we're working on something different?