DW> I've been reviewing the test failures from the gating job and what
DW> concerns me is that at least some of them are legitimate.
Yeah, indeed, I didn't mean to minimize the severity of the actual
issues, I just wonder if gating new tempest changes on whether or not
the suite runs "right now" makes total sense. For unit tests, it does,
but since tempest and nova are decoupled, it's a bit of a strange
relationship :)
In other suites, I've seen an XFAIL result used to mark tests that we
know are failing right now so that they're not SKIPped like tests that
are missing some component, but rather just not fatal to the task at
hand. Maybe something like that would be useful in tempest? If I found a
bug in Nova right now and wanted to get a test into tempest ASAP to poke
it, submitting as XFAIL would (a) not break Jenkins because the test
failed (as expected) and (b) raise a flag when the test started to pass
to make sure that it gets un-marked as XFAIL.
DW> They may not be caused by the patch at hand, but servers and volumes
DW> going into error status definitely signal issues, whether they be in
DW> code or environment. I don't have access to the Tempest CI
DW> environment so I don't have much insight into those issues
DW> specifically, though there might be some additional error checking
DW> that we can do to get more information on what is going wrong.
Yeah, I've been trying to reproduce the issues locally, as I'm happy to
fix them up if I can figure out what the problem is. However, I feel
like I'm flying blind a bit, without a view into the CI machine itself :)
DW> I'm doing what I can Dan to get your patches reviewed. The trick
DW> being that since there is a dependency chain between most of the
DW> commits, it adds a level of complexity. Jay, who's done most of the
DW> CI setup thus far, is out of country, so I'm trying to find other
DW> folks I can reach out to help stabilize the environment.
Yeah, where is that slacker? :)
We're very appreciative of the review you've already provided, as well
as your efforts to help move those forward. I didn't intend to come off
whiny, I just wanted to make sure that we don't get too far down the
rabbit hole, and certainly don't want to get ignored because of the big
red "-1" next to us :)
David Kranz and I have been chatting on IRC today about hopefully
marking the errant tests as skip so that we can move forward.
Thanks!
