openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Hi all,

Firstly, apologies to Sapan Kona. Sapan, I'm going to be using a recent 
proposed set of tests from you as an example in this mailing list post. 
There have been many similar patches from many authors proposed and 
already accepted into Tempest. I'm using Sapan's patch as an example, 
not singly him out.

OK, so while we've done a commendable job so far in adding a large 
*quantity* of tests to Tempest, one ramification of this increase in 
tests has been the large increase in runtime for the whole Tempest test 
suite. Nowadays, the whole suite is taking upwards of 45 minutes to 
complete -- and that's with nearly 70 skipped tests!

We need to be more careful about *how* we write tests in Tempest, and we 
need to be mindful of how long each test added takes to run and the 
resources each test creates/modifies/destroys.

In many cases, we've allowed tests to be added to Tempest that take a 
significant amount of time to run -- and these tests often can and 
should be combined into fewer test methods that use fewer resources.

I'm going to show an example below that demonstrates what we need to do 
when adding tests to Tempest. This example is from Sapan's recent 
proposed patch adding more tests for security groups:

https://review.openstack.org/#/c/8801/

The patch adds 4 new tests to the Compute API security groups test case. 
These tests check the following conditions:

1) Test that a security group can be added to a running server and 
removed from a running server
2) Test that a security group CANNOT be deleted if it is in use in an 
active server
3) Test that a security group can be deleted if it is not in use in any 
active servers
4) Test that a security group can be added to a server after it has been 
rebooted and removed from that server and that a security group CANNOT 
be deleted if it is in use in a server that has been rebooted

Unfortunately, because the patch adds four test methods, each of which 
create a server instance, the runtime is increased by more than 200 
seconds. However, all of the above checks can be done in a single test 
methods using a single server resource.

Here is the code that consolidates the four test methods into a single one:

http://paste.openstack.org/show/19062/

Note that all of the checks are asserted in the single test method, but 
only a single expensive server resource is created to assert these 
checks, not four.

And the runtime shows 3 less tests, but runs in 200 less seconds:

http://paste.openstack.org/show/19061/

We need to consider these types of things when we write tests for 
Tempest. The point should not be to just increase the *quantity* of 
tests, but rather to increase the testing while keeping the runtime of 
tempest down.

Thanks for listening,
-jay