openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Hi Yaniv, answers inline...

On 10/22/2012 11:41 AM, Yaniv Kaul wrote:
> On 10/22/2012 05:33 PM, Jay Pipes wrote:
>> Hi Sean :)
>>
>> Here's a quick recap:
>>
>> We agreed:
>>
>> * nosetests just isn't a good foundation for our work -- especially
>> regarding performance/parallelism
> 
> Any proposed alternatives?
> I am looking from time to time at
> https://code.google.com/p/robotframework/ and wondering if it allow
> faster test cases development

I don't like the behaviour in Robot Frameworks of creating test cases in
a tabular format:

http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7#id377

I vastly prefer code-based test cases.

The library we decided to take a look at was testtools, written in part
by Robert Collins, who is a member of the CI team working on OpenStack
stuff at HP:

http://testtools.readthedocs.org/en/latest/index.html

In the past we've also looked at PyVows:

http://heynemann.github.com/pyvows/

as well as DTest:

https://github.com/klmitch/dtest

Basically, the issues we have with nosetests are:

* It is very intrusive
* It doesn't handle module and class-level fixtures properly (see
https://github.com/nose-devs/nose/issues/551)
* Multiprocessing plugin is total fail
(https://github.com/nose-devs/nose/issues/550)

Anything that makes test code cleaner, with the ability to handle
fixtures cleanly, annotate dependencies between tests, and parallelize
execution effectively is fine in my book. :)

>> * We need to produce good, updated documentation on what different
>> categories of tests are -- smoke, fuzz, positive/negative, etc -- and
>> put this up on the wiki
> 
> Perhaps a naming convention to the test case names, such as
> component_category_test_case?
> For example:
> compute_sanity_launchVM()
> compute_negative_launchVM()

No, we use decorators to annotate different types of tests (grep for
@attr\(type= in Tempest. What we don't have is good basic documentation
on what we agree is an acceptance/smoke test and what isn't ;)

>> * We need to produce a template example test case for Tempest that
>> provides excellent code examples of how to create different tests in a
>> best practice way -- I believe David Kranz is going to work on this first?
>> * We need to get traceability matrixes done for the public APIs -- this
>> involves making a wiki page (or even something generated) that lists the
>> API calls and variations of those calls and whether or not they are
>> tested in Tempest
> 
> Wouldn't code coverage report be better?
> If you call FunctionX(param1, param2) - and you've called it with 2
> different param1 values and always the default in param2 - what does it
> mean, from coverage perspective?

No, unfortunately code coverage doesn't work for functional integration
tests in the same way it does for unit tests, for a number of reasons:

1) Tempest executes a series of HTTP calls against public REST endpoints
in OpenStack. It has no way of determining what code was run **on the
server**. It only has the ability to know what Tempest itself executed,
not what percentage of the total API those calls represented

2) Specs don't always exist for the APIs -- yes, I know, this isn't
good. Especially problematic are some of the Compute API extensions that
aren't documented well, or at all.

Best,
-jay

> Thanks,
> Y.
> 
>> * I will start the traceability matrix stuff and publish for people to
>> go and update
>> * Antoni from HP is going to investigate using things in testtools and
>> testrepository for handling module and package-level fixtures and
>> removing some of the nose-based cruft
>> * I will be working on the YAML file that describes a test environment
>> so that different teams that use different deployment frameworks can use
>> a commonly-agreed format to describe their envs to the CI system
>> * A new member of Gigi's team (really sorry, I've forgotten your name!
>> :( ) is going to look at the fuzz testing discussion from earlier and
>> see about prototyping something together that would be used for negative
>> and security/fuzz testing -- this would enable us to remove the static
>> negative tests from Tempest's main test directories. For the record (and
>> for the team member whose name I have forgotten, here is the relevant
>> link: https://lists.launchpad.net/openstack-qa-team/msg00155.html and
>> https://lists.launchpad.net/openstack-qa-team/msg00156.html)
>>
>> Best,
>> -jay
>>
>> On 10/19/2012 02:47 PM, Sean Gallagher wrote:
>>> Daryl,
>>> I wasn't able to make the follow-up meeting. :/
>>>
>>> Can you/ someone send out a recap?
>>>
>>> David: you had a list of items. Can you post or share that somewhere?
>>>
>>> We discussed Blueprints for managing some of the planning work.
>>>
>>> What about higher level planning docs? Useful? Do we have any? Should we?
>>>
>>> Re Google Hangout next week, I'm interested.
>>>
>>> -sean
>>>
>