openstack team mailing list archive

["Mark Washenberger" <mark.washenberger@xxxxxxxxxxxxx>]

I think it is commendable to identify bugs even if you can't fix them at the time. I hope that we don't create incentives to ignore bugs you find during development just to get your own merge through.

But I'm worried about staleness and usefulness with known bugs. If the known bugs test cases aren't actually running, they are basically just cosmetic. The code base can drift from them with absolutely no consequence.

Even if we run the known bug test cases and ensure that they fail, there is a lot of room for divergence. Usually a good unit test has a relatively small number of ways it can succeed, and a large number of ways it can fail. If we are just ensuring failure, what's to prevent the code base from drifting in a way that causes the test to fail in a different way?

To fix this last problem, known bugs test cases would actually have to become specific passing characterization tests that document undesirable behavior. At this point, the tests are no longer really useful when it comes time to fix the bug.

So I guess if you have a useful test case for a bug, just stick it in the bug report on launchpad if you are unable to fix it in a timely fashion or find someone to help you fix it.

Just my 2 cents, thanks.

"Dan Prince" <dan.prince@xxxxxxxxxxxxx> said:

> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> I'm not a big fan of 'known bugs' in unit tests. Unit tests should always pass.
> How practical is it that I'm going to invest the time to write a unit tests on a
> bug which I'm then not able to fix in the same merge. In many cases writing the
> test cases are actually harder than writing the code to fix the actual bug.
>  
> If you really need to write a test case ahead of time (perhaps even to make your
> case that a bug exists) why not just create a launchpad bug and then attach the
> test case as a patch to the bug report?
>  
> Seems like 'know bugs' also provides a mechanism for potentially stale unit tests
> code to hide out in the nova codebase? If code (including test code) is being
> actively used then I'd actually prefer to not have it be in the codebase.
>  
> 
> Lastly, QA testers would probably focus more on the functional and integration
> types of testing rather than unit tests right?
>  
> Dan
>  
> -----Original Message-----
> From: "Justin Santa Barbara" <justin@xxxxxxxxxxxx>
> Sent: Monday, February 28, 2011 1:56pm
> To: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: [Openstack] How to deal with 'tangential' bugs?
> 
> 
> Jay and I have been having an interesting discussion about how to deal with bugs
> that mean that unit tests _should_ fail.  So, if I find a bug, I should write
> a failing unit test first, and fix it (in one merge).  However, if I can't
> fix it, I can't get a failing unit test merged into the trunk (because it fails).
>  It may be that I can't get what I'm actually working on merged with good
> unit tests until this 'tangential' bug is fixed.
> (The discussion is
> here: [https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227]
> https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227)
> I suggested that we introduce a "known_bugs" collection.  It would have a set
> of values to indicate bugs that are known but not yet fixed.  Ideally these
> would be linked to bug reports (we could mandate this).  When a developer
> wants to write a test or behavior to work around a particular bug, they can
> control it based on testing this collection ("if 'bug12345' in known_bugs:")
>  When someone is ready to fix the bug, they remove the bug from the
> collection, the unit tests then fail, they fix the code and commit with the
> known_bugs item removed.
> This would let people that find bugs but can't or don't want to fix them still
> contribute unit tests.  This could be a QA person that can write tests but
> not necessarily code the fix.  This could be a developer who simply isn't
> familiar with the particular system.  Or it could be where the fix needs to
> go through the OpenStack discussion process.  Or it could simply be a
> train-of-thought / 'flow' issue.
> Take, for example, my favorite OpenStack API authentication issue.  To get a
> passing unit test with OpenStack authentication, my best bet is to set all three
> values (username, api_key, api_secret) to the same value.  This, however, is
> a truly terrible test case.  Having "known_bugs" marks my unit test as being
> suboptimal; it lets me provide better code in the same place (controlled by the
> known_bugs setting); and when the bug fixer comes to fix it they easily get a
> failing unit test that they can use for TDD.
> Jay (correctly) points out that this is complicated; can cause problems down the
> line when the bug is fixed in an unexpected way; that known_bugs should always be
> empty; and that the right thing to do is to fix the bug or get it fixed.  I
> agree, but I don't think that getting the bug fixed before proceeding is realistic
> in a project with as many stakeholders as OpenStack has.
> Can we resolve the dilemma?  How should we proceed when we find a bug but
> we're working on something different?
> Justin
>