openstack-qa-team team mailing list archive

[Dan Smith <danms@xxxxxxxxxx>]

JB> 1) We designed the devstack-gate with a facility to install the SSH
JB> key of the developer whose change failed onto the VM and give it to
JB> them for debugging purposes -- however, we've yet to have a
JB> devstack-gate node provider give us permission to hand out VMs like
JB> that.  I'm sorry that hasn't happened, though in the mean time, if
JB> there's any useful information (logs, output of ps or ip commands,
JB> etc) you'd like to be copied off of the machines that we aren't
JB> already doing, please let me know (or submit a patch to
JB> openstack-ci/devstack-gate).

Oh, that'd be sweet. I think I'd take Jenkins' name in vain a lot less
if I could get in and check things out like that. Thanks for aiming this
high, despite the obvious problem it generates.

JB> But as we get to the point where the non-smoke tests are failing due
JB> to real problems with the core projects rather than tempest itself,
JB> we should look at making those tests part of the wider gate (either
JB> by making them smoke tests, or expanding the gate to run more than
JB> just the smoke tests for all projects).

Perhaps changing the cut point for what is considered a smoke test to
include anything that isn't likely to be fragile, or adding a third
grouping that includes a wider selection of tests would be
appropriate. The imbalance between tempest and the other projects is
(IMHO) is the actual problem, as you identified, not just that tempest
is gated on the full set of course.

JB> Of course, run time is a consideration, but we wrote Zuul largely to
JB> deal with this problem -- Zuul performs gate tests in parallel (but
JB> still tests each change individually as it will be merged).  So
JB> while we definitely would like to keep run-time as short as
JB> possible, running the Jenkins jobs in parallel means we don't have
JB> to wait for each change to be tested in series.  So in short, do
JB> please make tempest run as fast as possible, but we want to run
JB> useful tests, which takes time, and that's something we're prepared
JB> to deal with.

One thing I noticed while doing some of this was that Jenkins doesn't
seem to know about dependent patches, and will often run them in some
sub-optimal order. This means that if the third patch in a series of ten
is broken, it still runs everything ten times, instead of saying "well,
patch 3/10 is broken, so 4/10 and beyond will be skipped." Not sure how
hard that logic would be to integrate, but it might help to cut down
some load. I was generating ten test runs every hour or so the other day
trying to diagnose what was going on... :D

Thanks!

-- 
Dan Smith
IBM Linux Technology Center