| Thread Previous • Date Previous • Date Next • Thread Next |
On 10/5/2012 3:16 PM, Yaniv Kaul wrote:
On 10/05/2012 07:50 PM, David Kranz wrote:A brief status update:1. The fixes for the full tempest gate have been merged. Tempest should run without errors except that there is still a flakey volume test https://bugs.launchpad.net/tempest/+bug/1056213. This is not a bug in tempest and we are struggling with what to do about it. Any ideas would be very welcome.Continue executing, so people will be aware of the issue, and perhaps be able to triage to more specific kernel versions / distributions. (The fix proposed in https://bugs.launchpad.net/cinder/+bug/1023755/comments/22 - to not zero out on volume deletion, isn't the greatest from security perspective, but that's a different matter).2. There is now a stable/folsom branch for tempest. If you are running tempest against a folsom clusteryou should now be using that branch of tempest.Is there a commit policy to this branch? what about fixes from master?
We will discuss this at the next QA meeting. Obviously it has to track stable/folsom for the other projects but whether to backport new tests from master is a matter of resources. For previous stable branches the
policy was that if interested parties wanted to backport fixes they could.
3. After discussion with the ci team, we are going to start gating all core project checkins on full tempest minus flakey tests and "non-gating" tests.I hope this will happen next week.What's the definition of flaky?
For this purpose, flaky means any test that should always succeed but fails sometimes in a non-reproducible way. For example, the volume test mentioned above has been failing perhaps 5-10% of the time. That may be acceptable for tempest checkins, but not for other projects. Any external gating on projects has to be rock solid.
4. We will be starting a nightly run of full tempest plus "non-gating" tests. We will be adding stress, configuration and performancetests to the non-gating suite in the coming weeks.Great to hear, I hope we'll be doing the same - lets try to coordinate those efforts.
Indeed. There is a session in the QA track at the summit titled "Bringing OpenStack QA into the Open": http://summit.openstack.org/cfp/details/81 The exact purpose of this session is to encourage more open development and execution of tests among the various OpenStack contributors. I am hoping that people involved in testing from various companies will attend the session.
-David
| Thread Previous • Date Previous • Date Next • Thread Next |
