launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10-12-14 06:17 AM, Julian Edwards wrote:
> On Monday 13 December 2010 17:15:50 Aaron Bentley wrote:
>> On 10-12-13 11:05 AM, Julian Edwards wrote:
>>> I think most of us have a pretty good idea of what constitutes a slow
>>> test now, and where the time is lost in those.  I don't think you need
>>> absolute numbers.
>>
>> I think that we'll have a better idea if we articulate our goals.  I
>> think that reasonable people could disagree over whether a 1-second test
>> is too slow.
> 
> Most reasonable people will use common sense here, and only unreasonable 
> people will disagree.  Or more to the point, I'd expect the people involved to 
> have a short, rational discussion and come to a quick, mature, informed 
> decision.

I still feel like I don't have enough information to make an informed
decision.

>> Those are all great suggestions for writing tests, but they don't
>> directly apply to a reviewer who is trying to asses whether tests are
>> too slow.
> 
> I'm not sure if you're joking or not here.  These are all points that a 
> reviewer can apply when reviewing test code to see if it could be quicker.

They might make good best practices, but they are not a metric.
Implementing all of them may not appreciably speed up a test.  Maybe the
test is already blindingly fast.  Maybe the test set-up is slow.
Perhaps the code under test is slow.

We're talking about optimising tests, and "premature optimization is the
root of all evil".

Optimisation has costs-- it take developer effort and it tends to
produce less readable, less robust code.  Tweaking tests that have been
written using TDD also has costs, because TDD guarantees that the tests
will fail without the change, but test-after-coding does not.  Test
doubles are sometimes the best way to speed up a test, but this speed-up
comes at the cost of reduced realism.

Until you know that code is slow, you don't know that the benefits of
optimization will outweigh the costs.  Robert and Brad both recommended
timing tests as the way to determine whether they're fast enough.

Once you know code is slow, you have to get an idea of why it's slow.
Next you change the code to address that issue.  Finally, you need to
measure whether your changes had the desired effect.  If the code is
still too slow, you'll need to pursue more changes.

Without measurement, optimisation is just cargo-culting.  It provides no
guarantee that your tests will be fast.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0HhqsACgkQ0F+nu1YWqI2GWwCfVNL+Fzz+b/kV92qkwfZLvsEl
68EAnRd7O5BJ6+Ruekx4+wbyYE1LleEv
=OUKv
-----END PGP SIGNATURE-----