launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Summary:

 * If you are writing code in a test to wait for something to happen in
wall-clock time...

   * make sure that you are only doing this in setup or teardown, to
prevent a permanent hang in the test suite; and

   * make sure that you provide a really generous amount of time--say,
10 or 20 times what you seem to require locally--before you make the
test give up and fail.

Negative corollaries:

 * Don't write tests that use wall-clock time to actually verify the
performance of anything.

 * Don't be stingy with the wall-clock time before your test gives up on
some setup or teardown happening.  Give it *plenty* of time.

Rationale:

 * Parallel tests stress system resources a *lot* more than we are used
to.  Specifically, production buildbot slaves scheduled to be purchased
will run *24* test systems in parallel on the same machine (with
parallel postgres instances, memcaches, rabbitmqs, and so on).  This
introduces a whole lot of variability in terms of resource contention.
We're going from barely being able to tax a two core machine to really
pushing a 12 core hyperthreaded machine.

 * Using wall-clock time to verify something is fragile anyway.  Test
something else related instead (e.g., the number of queries performed).

Thanks for your attention.  Let me know of your concerns.

Gary