openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Sorry for late reply... in Bulgaria on vacation until 29th... Comments
inline.

On 08/16/2012 08:13 PM, David Kranz wrote:
> I think this clearly illustrates why it is so important for QA to be 
> done more in the open, and to post tempest blueprints for largish 
> projects like this. I have a theory that one important reason so many 
> organizations choose not to do this is that once they submit their tests 
> to the public repository (in this case tempest), they have to get every 
> change approved. I had the same issue when we submitted our stress 
> tests. There cannot be any advantage to doing qa in private. We need to 
> make submitting test code free of such barriers. More tests can only be 
> good.

Agreed.

> I propose that we set up a 'contrib' part of tempest that is exempt from 
> core review.  At some point tests can be moved into tempest proper 
> though, as a gating job, there is a limit to what can be there due to 
> runtime considerations. But we already have a lot of tests running every 
> night distributed across organizations. Why not post the code so others
> can share and improve it? We would need to:
> 
> 1. Set up the contrib directory
> 2. Establish some guidelines for submitting changes to other people's 
> contributions
> 3. Establish the nightly build for contributed tests, probably including 
> running some outside of jenkins

I am not really supportive of this idea. I don't think there should be
any code that is exempt from code review, period. I believe there just
needs to be a better focus on doing everything upstream *first* and not
getting into situations where you are developing code or tests and then
"push it upstream". I just don't see the point of doing that; it just
doesn't make much sense to me.

It's the same argument for why we don't accept patches to core projects
against stable branches before the patch has been accepted into
HEAD/trunk. We want people working on the main line, not in some
walled-off implementation. Same goes for Tempest IMHO.

> I am not sure what review flexibility is afforded by gerrit that would 
> help or hinder this effort.

Again, I'm opposed to changing the (in)flexibility of code reviews. Code
review is there for a reason, and if "private test code" wouldn't pass
code review for some reason, there's a problem with it and it should not
go into Tempest. I personally think the only valid reason to have
private test code is to test functionality that does not exist in the
upstream projects. And for that reason, it doesn't belong in the
upstream Tempest project anyway...

Best,
-jay