openstack-qa-team team mailing list archive

[Daryl Walleck <daryl.walleck@xxxxxxxxxxxxx>]

I think Jay mostly covered things, but here's the notes I took for this meeting:


Goals for Grizzly

  *   Implement an efficient, scalable parallelization solution
  *   Work with other teams (Horizon, Red Dwarf, etc) to bring representation from all projects into Tempest
  *   As contributors, bring all existing functional/stress/benchmarking tests into the overall Tempest suite
  *   Craft and deploy a more acceptable set of gating tests
  *   Refactoring for cleanup after large scale merge from groups

Action Items

  *   Remove dependencies on Nose/provide merge-prop of what Tempest would look like using testtools/testrepository
  *   Profile current tests to look for bottlenecks (dwalleck)
  *   Propose blueprints for all test sets not in trunk Tempest (Daryl for Rack, ? for others)
  *   Determine most critical scenarios and combine as many tests as we can logically to make this gate set of tests

________________________________
From: openstack-qa-team-bounces+daryl.walleck=rackspace.com@xxxxxxxxxxxxxxxxxxx [openstack-qa-team-bounces+daryl.walleck=rackspace.com@xxxxxxxxxxxxxxxxxxx] on behalf of Yaniv Kaul [ykaul@xxxxxxxxxx]
Sent: Monday, October 22, 2012 10:41 AM
To: Jay Pipes
Cc: Rick Lopez; openstack-qa-team@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack-qa-team] Moving follow-up Unconference to 1:45 today

On 10/22/2012 05:33 PM, Jay Pipes wrote:

Hi Sean :)

Here's a quick recap:

We agreed:

* nosetests just isn't a good foundation for our work -- especially
regarding performance/parallelism

Any proposed alternatives?
I am looking from time to time at https://code.google.com/p/robotframework/ and wondering if it allow faster test cases development


* We need to produce good, updated documentation on what different
categories of tests are -- smoke, fuzz, positive/negative, etc -- and
put this up on the wiki

Perhaps a naming convention to the test case names, such as component_category_test_case?
For example:
compute_sanity_launchVM()
compute_negative_launchVM()


* We need to produce a template example test case for Tempest that
provides excellent code examples of how to create different tests in a
best practice way -- I believe David Kranz is going to work on this first?
* We need to get traceability matrixes done for the public APIs -- this
involves making a wiki page (or even something generated) that lists the
API calls and variations of those calls and whether or not they are
tested in Tempest

Wouldn't code coverage report be better?
If you call FunctionX(param1, param2) - and you've called it with 2 different param1 values and always the default in param2 - what does it mean, from coverage perspective?

Thanks,
Y.


* I will start the traceability matrix stuff and publish for people to
go and update
* Antoni from HP is going to investigate using things in testtools and
testrepository for handling module and package-level fixtures and
removing some of the nose-based cruft
* I will be working on the YAML file that describes a test environment
so that different teams that use different deployment frameworks can use
a commonly-agreed format to describe their envs to the CI system
* A new member of Gigi's team (really sorry, I've forgotten your name!
:( ) is going to look at the fuzz testing discussion from earlier and
see about prototyping something together that would be used for negative
and security/fuzz testing -- this would enable us to remove the static
negative tests from Tempest's main test directories. For the record (and
for the team member whose name I have forgotten, here is the relevant
link: https://lists.launchpad.net/openstack-qa-team/msg00155.html and
https://lists.launchpad.net/openstack-qa-team/msg00156.html)

Best,
-jay

On 10/19/2012 02:47 PM, Sean Gallagher wrote:


Daryl,
I wasn't able to make the follow-up meeting. :/

Can you/ someone send out a recap?

David: you had a list of items. Can you post or share that somewhere?

We discussed Blueprints for managing some of the planning work.

What about higher level planning docs? Useful? Do we have any? Should we?

Re Google Hangout next week, I'm interested.

-sean