openstack-qa-team team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On 05/04/2012 09:28 AM, David Kranz wrote:
> On 5/4/2012 9:08 AM, Jay Pipes wrote:
> > On 05/04/2012 06:14 AM, Karajgi, Rohit wrote:
> > > Hi,
> > >
> > > What is the policy that we should or are following for test cases that
> > > fail due to an existing Open Bug in Launchpad?
> > >
> > > For eg:
> > > https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64
> > >
> > > skips the test and posts the Bug ID in the message.
> > >
> > > Do we submit the test for review with the @skip(until Bug #xyz is fixed)
> > > decorator applied?
> >
> > This is the choice that I believe is the easiest and simplest. We just
> > need to be vigilant to remove the skips when/if the bug is fixed.
> >
> > -jay
> Jay, I agree with the idea that we should check the test in anyway. But
> we have the issue of who remembers to remove the skip decorator when the
> bug is fixed. Since we are going to be running tempest nightly, I think
> it might be better to make these tests fail in the absence of the bug
> rather than being skipped. I think there may already be some cases in
> Tempest that do this.

Well, we're going to be running Tempest nightly, but also a subset for 
every commit into any of the core projects... and that subset will be a 
full merge gate.

So, a further question to answer is whether we treat *some* of the tests 
differently than others with respect to how skips are handled?

Best,
-jay