← Back to team overview

openstack-qa-team team mailing list archive

Re: [Openstack] [QA] Aligning "smoke" / "acceptance" / "promotion" test efforts

 

On 05/03/2012 03:54 PM, Daryl Walleck wrote:
So my first question is around this. So is the claim is that the client
tools are the default interface for the applications?

Sorry, perhaps a better term would have been "the most common interface to OpenStack Compute"...

>  While that works
for coders in python, what about people using other languages? Even
then, there's no guarantee that the clients in different languages are
implemented in the same way. Tempest was designed originally because
while it does use an abstraction between the API and the tests, there is
nothing to "assist" the user by retrying and the like. While I think
there's a place for writing tests using the command line clients, to me
that would be a smoke test of a client and not as much a smoke test of
the API.

I understand your point, and I want to make it clear that I'm not advocating for getting rid of the Tempest REST client at all. I'm only advocating that we have a set of tests -- smoke tests -- that use the Python novaclient library, run a small subset of the total tests, and exercise only the most common and simple use cases for OpenStack.

Here is where I think a sensible breakdown is:

* "Smoke tests"

 - Test ONLY *simple*, *fast* code paths for common use cases
 - Use a Manager class that has a novaclient Client object

* Negative tests

- Test all code paths that should produce errors or negative results (NotFound, etc)
 - Use a Manager class that has the Tempest REST client

* Fuzz tests

 - Test all code paths with randomized input data
 - Use a Manager class that has the Tempest REST client

* Advanced tests

- Test extensions, interaction between multiple components, advanced features of the API, etc - Use a Manager class that has a novaclient Client object OR the Tempest REST client, depending on whether:
  - novaclient actually has the ability to perform the necessary actions
- If the test needs to poke at the API in a way that novaclient would interfere with -- and thus the Tempest REST client would be ideal

Best,
-jay


References