openstack team mailing list archive

["Daniel P. Berrange" <berrange@xxxxxxxxxx>]

On Tue, Jul 03, 2012 at 04:48:32PM -0700, James E. Blair wrote:
> Hi,
> 
> As mentioned in the thread "Jenkins and transient failures", we've had
> an unusually high number of transient failures in Jenkins lately.  We've
> done several things in response to that:
> 
> 1) Monty identified a problem with our pypi mirror which was the cause
> of many of the errors, and corrected it.
> 
> 2) Monty is continuing to work on the single dependency list which
> should allow us to switch to using our local pypi mirror exclusively,
> further reducing transient network errors, as well as significantly
> speeding up test run time.
> 
> 3) Several transient errors were caused by failed fetches from Gerrit.
> While consulting with the Gerrit authors about tuning, they discovered a
> bug in Gerrit where a 5 minute timeout was being interpreted as a 5
> millisecond timeout.  I have updated our gerrit configuration to work
> around that.
> 
> 4) Clark Boylan implemented automatic retrying for the git fetches that
> we use with Jenkins.
> 
> 
> I hope that we'll get to the point where we have almost no transient
> network errors when testing, but we know it will never be perfect, so at
> the CI meeting we discussed how best to implement retriggering with
> Zuul.  Clark added a comment filter that will retrigger Jenkins if you
> leave a comment that matches a regex.
> 
> We currently run two kinds of jobs in Jenkins, the check job and the
> gate job.  The check jobs run immediately when a patchset is uploaded
> and vote +/-1.  The gate jobs run on approval, queue up across all
> projects and vote +/-2 (if they fail, jobs behind them in the gate
> pipeline may need to run again).
> 
> 
> To retrigger the initial Jenkins check job, just leave a comment on the
> review in Gerrit with only the text "recheck".
> 
> To retrigger the Jenkins merge gate job, leave a comment with only the
> text "reverify", or if you are a core reviewer, just leave another
> "Approved" vote.  (Don't leave a "reverify" comment if the change hasn't
> been approved yet, it still won't be merged and will slow Jenkins down.)

Thanks to the team who worked on resolving the problems from last &
also adding this retrigger support. The latter will bring a nice
improvement to productivity should there be problems in the future.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|