openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

QAers,

There has been a pile-up of code reviews recently, and I wanted to 
explain a decision that a number of core QA team members reached last 
week and why some patchsets have not been reviewed.

Two of the goals of Tempest are to have a functional integration test 
suite that stresses *different* things than the unit tests and runs in a 
reasonable amount of time (< 20 minutes or so).

While it's been great to see the large influx of negative tests added 
recently to Tempest, there is a concern that the incremental value these 
tests might add does not counterbalance the lengthy times that some of 
the tests take to run.

It was the advice of these core team members (including myself), that we 
put a moratorium on adding new negative tests to Tempest at this point 
and look instead at doing the following:

* Determine which negative test cases that are currently in Tempest DO 
NOT exist in the unit test suite of the corresponding core project, add 
those unit tests appropriately, and then remove the negative tests from 
Tempest
* Use a fuzz-testing grammar-based tool such as randgen [1] to do 
negative testing of the APIs [2]
* Focus all energies on writing test cases that stress the integration 
test points between the services and do more than just API-level 
validation -- in other words, doing more SSH-ing into an instance for 
verification of networking setups, init files, etc
* Change the way we are currently using Launchpad's bugs and blueprints 
system in the following ways:
 (1) No longer have bugs that stay around forever, with lists of new 
small tests to be added. Instead, bugs should describe a very specific 
task, in detail, and should be closed once the test is added to Tempest
 (2) Have someone dedicated to Bug Triaging and cleaning up the current 
list of open bugs -- I believe David Kranz and I will be tackling this.

Please respond to this post if you have any questions about the 
important points above. We need to refocus the QA team a bit to add 
greater value and make Tempest the integration test suite that the other 
core projects can rely on.

All the best,
-jay

[1] https://launchpad.net/randgen
[2] https://lists.launchpad.net/openstack-qa-team/msg00155.html