openstack-qa-team team mailing list archive

[Lorin Hochstein <lorin@xxxxxxxxxxxxxxxxxx>]

On Jun 29, 2012, at 3:27 PM, Jay Pipes wrote:

> Hi all,
> 
> OK, so while we've done a commendable job so far in adding a large *quantity* of tests to Tempest, one ramification of this increase in tests has been the large increase in runtime for the whole Tempest test suite. Nowadays, the whole suite is taking upwards of 45 minutes to complete -- and that's with nearly 70 skipped tests!
> 
> We need to be more careful about *how* we write tests in Tempest, and we need to be mindful of how long each test added takes to run and the resources each test creates/modifies/destroys.
> 
> In many cases, we've allowed tests to be added to Tempest that take a significant amount of time to run -- and these tests often can and should be combined into fewer test methods that use fewer resources.
> 

[snip]

> We need to consider these types of things when we write tests for Tempest. The point should not be to just increase the *quantity* of tests, but rather to increase the testing while keeping the runtime of tempest down.
> 


As it happens, the problem of reducing regression testing time (by reducing the number of tests in an intelligent way) is one of the most heavily studied areas in software engineering research.

I don't have any personal expertise in this area, but if you do a search for "regression test selection" on Google Scholar, you'll find a ton of stuff on this. Gregg Rothermel at University of Nebraska-Lincoln is one big name in this area.

Take care,

Lorin
--
Lorin Hochstein
Lead Architect - Cloud Services
Nimbis Services, Inc.
www.nimbisservices.com