openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On 05/03/2012 03:54 PM, Daryl Walleck wrote:
> So my first question is around this. So is the claim is that the client
> tools are the default interface for the applications?

Sorry, perhaps a better term would have been "the most common interface 
to OpenStack Compute"...

>  While that works
> for coders in python, what about people using other languages? Even
> then, there's no guarantee that the clients in different languages are
> implemented in the same way. Tempest was designed originally because
> while it does use an abstraction between the API and the tests, there is
> nothing to "assist" the user by retrying and the like. While I think
> there's a place for writing tests using the command line clients, to me
> that would be a smoke test of a client and not as much a smoke test of
> the API.

I understand your point, and I want to make it clear that I'm not 
advocating for getting rid of the Tempest REST client at all. I'm only 
advocating that we have a set of tests -- smoke tests -- that use the 
Python novaclient library, run a small subset of the total tests, and 
exercise only the most common and simple use cases for OpenStack.

Here is where I think a sensible breakdown is:

* "Smoke tests"

 - Test ONLY *simple*, *fast* code paths for common use cases
 - Use a Manager class that has a novaclient Client object

* Negative tests

 - Test all code paths that should produce errors or negative results 
(NotFound, etc)
 - Use a Manager class that has the Tempest REST client

* Fuzz tests

 - Test all code paths with randomized input data
 - Use a Manager class that has the Tempest REST client

* Advanced tests

 - Test extensions, interaction between multiple components, advanced 
features of the API, etc
 - Use a Manager class that has a novaclient Client object OR the 
Tempest REST client, depending on whether:
  - novaclient actually has the ability to perform the necessary actions
  - If the test needs to poke at the API in a way that novaclient would 
interfere with -- and thus the Tempest REST client would be ideal

Best,
-jay