openstack-qa-team team mailing list archive

[corvus@xxxxxxxxxxxx (James E. Blair)]

Dan Smith <danms@xxxxxxxxxx> writes:

> One thing I noticed while doing some of this was that Jenkins doesn't
> seem to know about dependent patches, and will often run them in some
> sub-optimal order. This means that if the third patch in a series of ten
> is broken, it still runs everything ten times, instead of saying "well,
> patch 3/10 is broken, so 4/10 and beyond will be skipped." Not sure how
> hard that logic would be to integrate, but it might help to cut down
> some load. I was generating ten test runs every hour or so the other day
> trying to diagnose what was going on... :D

Since about the beginning of August, Zuul has been testing dependent
changes in the correct order, but you have described a situation which
could be optimized further.

Here's what we are doing:

* Dependent patch series are tested in the correct order (regardless of
  the order in which individual patches are approved, and patches are
  only queued for testing once all the patches ahead of them are
  approved).  This eliminated a huge amount of churn, as you can
  imagine.

* When a patch fails a test, we immediately cancel tests for changes
  behind it to begin recovering resources (but we run the tests on the
  patch at the front of the queue to completion so it has a full report
  of the failure).  When that's done, we restart the tests for changes
  behind it, but without the assumption that patch merged.

That's where the madness you described comes in -- that makes perfect
sense if the queue is built up of independent changes -- but if the
queue is composed of dependent changes which can't actually merge if the
one ahead didn't, you're right, we should immediately stop testing and
dequeue all of them (perhaps leaving a note in the review saying testing
was aborted due to the failure of a needed change).

Thanks for bringing that up, that's a pretty good optimization that we
should be able to make without too much trouble.

BTW, you can see Zuul's view of the queues here:

  https://jenkins.openstack.org/zuul/status

-Jim