openstack team mailing list archive

["Karajgi, Rohit" <Rohit.Karajgi@xxxxxxxxxxx>]

If the approach would be to use the client tools to "only smoke test" the Openstack API, I would somewhat
agree. I think the smoke tests should be shallow but wide, covering every API in OpenStack but not ever 'scenario'.
So to me a "subset of test cases" would be a subset of "test cases from every OpenStack API"  that would make a smoke suite.

If the default client tools are ahead of the Tempest Rest client library with regard to API coverage, then it would make sense to use them
to quickly build a smoke test suite, IMHO.
The client managers should be used only to contribute to smoke tests, and not used randomly for more detailed scenario tests.

I'd agree with Daryl in saying that we should avoid Tempest to smoke test or any test the client tools, rather use them to smoke test
the API. The client tools should have their own tests.

-Rohit

From: openstack-bounces+rohit.karajgi=nttdata.com@xxxxxxxxxxxxxxxxxxx [mailto:openstack-bounces+rohit.karajgi=nttdata.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Daryl Walleck
Sent: Friday, May 04, 2012 1:24 AM
To: Jay Pipes
Cc: openstack-qa-team@xxxxxxxxxxxxxxxxxxx; <openstack@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [Openstack] [QA] Aligning "smoke" / "acceptance" / "promotion" test efforts

So my first question is around this. So is the claim is that the client tools are the default interface for the applications? While that works for coders in python, what about people using other languages? Even then, there's no guarantee that the clients in different languages are implemented in the same way. Tempest was designed originally because while it does use an abstraction between the API and the tests, there is nothing to "assist" the user by retrying and the like. While I think there's a place for writing tests using the command line clients, to me that would be a smoke test of a client and not as much a smoke test of the API.

Daryl

On May 3, 2012, at 12:01 PM, Jay Pipes wrote:


However, before this can happen, a number of improvements need to be made to Tempest. The issue with the "smoke tests" in Tempest is that they aren't really smoke tests. They do not use the default client tools (like novaclient, keystoneclient, etc) and are not annotated consistently.


______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data.  If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding