launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Hi all.  Gavin brought up a good question in a MP (https://code.edge.launchpad.net/~leonardr/lazr.authentication/initial-implementation/+merge/12989 
) and I wanted to highlight the discussion to the list.
The question is whether packages we release should include their test  
dependencies, even if we do not believe them to be necessary for the  
packages' operation.  For instance, should setup.py for  
lazr.authentication specify the testing tool wsgi_intercept as an  
installation requirement, even if it is only a testing tool?
Gavin felt we shouldn't, and I felt we should.  Sounds like it's time  
for a discussion. :-)  The following is my argument, ripped from the MP.
There's a tension between wanting to reduce dependencies for your  
users, and wanting to be confident that your tests actually reflect  
the performance of your code.  The second consideration is sometimes  
called "test what you fly, fly what you test," and is associated in  
the States with NASA, our space agency.  I have seen well-tested code  
that directly or indirectly depended on a package that was expected to  
only be a test dependency, but was actually a run-time dependency.   
Because the automated tests relied on the test dependencies to run,  
the only way to catch these kinds of packaging errors was to do manual  
QA that effectively duplicates the role of your automated tests.  I  
don't think that's a good way to release packages: I want to rely on  
my automated tests.
Therefore, I think that the "test what you fly, fly what you test"  
axiom should be followed as a rule, unless there are really compelling  
reasons not to.  A few additional dependencies are generally worth it  
to me for the additional confidence in my releases.
I do think that there is a subjective point at which the benefit  
outweighs the cost.  I probably start to feel that I can break the  
rule for library packages when the test dependencies are hard to build  
or have a count of more than five or six.  On the other hand, I pretty  
much never want to break the rule for deployments of production  
systems.  In other words, if lazr.authentication started getting  
really bloated because of test dependencies, I would be tempted to  
break the rule for its release, and separate out the test  
dependencies; contrariwise, if Launchpad got a lot of test  
dependencies, I would not be tempted at all to break the rule for its  
deployment.
Perhaps this is worth a reviewers' meeting discussion, or maybe we can  
reach consensus on the list.
Gary