← Back to team overview

launchpad-dev team mailing list archive

Please be very lenient in your test timeouts--or don't write them!



 * If you are writing code in a test to wait for something to happen in
wall-clock time...

   * make sure that you are only doing this in setup or teardown, to
prevent a permanent hang in the test suite; and

   * make sure that you provide a really generous amount of time--say,
10 or 20 times what you seem to require locally--before you make the
test give up and fail.

Negative corollaries:

 * Don't write tests that use wall-clock time to actually verify the
performance of anything.

 * Don't be stingy with the wall-clock time before your test gives up on
some setup or teardown happening.  Give it *plenty* of time.


 * Parallel tests stress system resources a *lot* more than we are used
to.  Specifically, production buildbot slaves scheduled to be purchased
will run *24* test systems in parallel on the same machine (with
parallel postgres instances, memcaches, rabbitmqs, and so on).  This
introduces a whole lot of variability in terms of resource contention.
We're going from barely being able to tax a two core machine to really
pushing a 12 core hyperthreaded machine.

 * Using wall-clock time to verify something is fragile anyway.  Test
something else related instead (e.g., the number of queries performed).

Thanks for your attention.  Let me know of your concerns.


Follow ups