On 06/28/2012 01:49 PM, Dan Prince wrote:
----- Original Message -----
From: "Monty Taylor" <mordred@xxxxxxxxxxxx> To:
openstack@xxxxxxxxxxxxxxxxxxx Sent: Thursday, June 28, 2012
11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests &
Gerrit merge blockers
On 06/28/2012 07:32 AM, Daniel P. Berrange wrote:
Today we face a situation where Nova GIT master fails to pass
all the libvirt test cases. This regression was accidentally
introduced by the following changeset
https://review.openstack.org/#/c/8778/
If you look at the history of that, the first SmokeStack test
run fails with some (presumably) transient errors, and added
negative karma to the change against patchset 2. If it were not
for this transient failure, it should have shown the regression
in the libvirt test case. The libvirt test case in question was
one that is skipped, unless libvirt is actually present on the
host running the tests. SmokeStack had made sure the tests would
run on such a host.
There were then further patchsets uploaded, and patchset 4 was
approved for merge. Jenkins ran its gate jobs and these all
passed successfully. I am told that Jenkins will actually run
the unittests that are included in Nova, so I would have expected
it to see the flawed libvirt test case, but it didn't. I presume
therefore, that Jenkins is not running on a libvirt enabled
host.
Kind of - it's sadly more complex than that ...
The end result was that the broken changeset was merged to
master, which in turns means any other developers submitting
changes touching the libvirt area will get broken tests reported
that were not actually their own fault.
This leaves me with the following questions...
1. Why was the recorded failure from SmokeStack not considered to
be a blocker for the merge of the commit by Gerrit or Jenkins or
any of the reviewers ?
2. Why did SmokeStack not get re-triggered for the later patch
set revisions, before it was merged ?
The answer to 1 and 2 is largely the same - SmokeStack is a
community contributed resources and is not managed by the CI team.
Dan Prince does a great job with it, but it's not a resource that
we have the ability to fix should it start messing up, so we have
not granted it the permissions to file blocking votes.
I would add that if anyone else is interested in collaborating on
making SmokeStack better I'm more than happy to give access. Its all
open source and has been since Cactus.
As is now SmokeStack can can cast a -1 vote and hopefully this is
proving to be useful. I'm open to suggestions.
I think it's stellar!
The tests that smokestack runs could all be written such that they
are run by jenkins.
I actually put in quite a bit of work to maintain an openstack_vpc
job on Jenkins post-Cactus. When we talked about gating on this job
at the Diablo conference the idea didn't seem to get very far... I
kind of saw that as the end of the line for maintaining an
openstack_vpc job and eventually it went away. Not sure who deleted
it, but anyway.
The way I see it there is value in both testing systems. Rather than
complaining about why one system exists and/or doesn't port its tests
to the other.... why don't we build on each others strengths. Seeing
a green "verified +1" from both Jenkins and SmokeStack on a review
should be very encouraging... and if one of the two systems fails it
might require further investigation.
I completely agree with that. I'm still hoping we'll see more systems
from more people so that the set of combinations get larger.
I think also there's clearly value in running tests, like how SmokeStack
is doing right now, that aren't necessarily part of the gate, but which
pro-actively provide useful information to the reviewers.
The repos that run the jenkins tests are all in git and managed by
openstack's gerrit. If there are testing profiles that it runs that
we as a community value and want to see part of the gate, anyone is
welcome to port them.
3. Why did Jenkins not ensure that the tests were run on a
libvirt enabled host ?
This is a different, and slightly more complex. We run tests in
virtualenvs so that the process used to test the code can be
consistently duplicated by all of the developers in the project.
This is the reason that we no longer do ubuntu package creation as
part of the gate - turns out that's really hard for a developer
running on OSX to do locally on their laptop - and if Jenkins
reports an blocking error in a patch, we want a developer to be
able to reproduce the problem locally so that they can have a
chance at fixing it.
The ability for developers to test things locally is very important.
For that matter SmokeStack all started with a project called
openstack_vpc, a project to spin up groups of cloud servers installed
with the latest OpenStack code. A developer can use a project like
openstack_vpc to spin up a set of servers in the cloud which builds
and installs custom built packages for a set of Git URLs. So
essentially the underpinnings of SmokeStack *can* all be done from a
local machine just like they run from the UI.
There is also value in testing things differently in ways which may
not be easy for all developers to reproduce. Take XenServer for
example... not every developer has access to a machine which can spin
up a mini XenServer cloud. Is there value in running upstream tests
on XenServer? I think so...
What about running OpenStack with PostgreSQL and MySQL, Rabbit and
Qpid? What I'm trying to do with SmokeStack is add value to our
testing matrix so that some of the things we aren't automatically
testing elsewhere get some coverage.
++
Reproducability is important... but the way I see it if everyone
always ran tests with exactly the same flags, or in the same
environments we might not find some things.
And where a developer can't reproduce something locally what I've
done is give them direct access to a box running a SmokeStack job so
they can troubleshoot it directly.
Yup. We've done this with devstack tests in jenkins too. Super helpful
for some of those weird times...
Problem arise in paradise though. libvirt being one of them. It's
not possible to install libvirt into a virtualenv, because it's a
swig-based module built as part of the libvirt source itself. One
of the solutions to this is to allow the testing virtual
environments to use packages installed at the system level. We
suggested this a little while ago, but this was rejected by the
nova team who valued the benefit of having a restricted test run so
that we know we've got all of the depends properly specified.
To that end, after chatting with Brian Waldon, I put this up as a
possible next try:
https://review.openstack.org/#/c/8949/
Which adds an additional testing environment that has system
software enabled and also installs additional "optional" things.
With that environment, we should be able to run a jenkins gate that
tests things with full libvirt, and also tests the mysql upgrade
paths, without screwing our fine friends who run OSX.
Fundamentally though - we're at a point of trying to have our cake
and eat it too. Either we want comprehensive testing of all of the
unit tests, or we want to be careful about not making the test
environment to hard for a developer to exactly mimic.
I'm obviously on the side of having us have gating tests that some
devs might not be able to do on their laptops - such as running
the libvirt tests properly. We're working on cloud software - worst
case scenario if there's an intractable problem, as dev can always
spin up an ubuntu image somewhere.
Obviously this was all made worse by the transient problems
we've had with the tests suite infrastructure these past 2 days,
but regardless it seems like we have a gap in our merge approval
procedures here.
IMHO, either SmokeStack needs to be made compulsory, or Jenkins
needs to ensure tests are run on suitable hosts like SmokeStack
does, or both.
The second is much more possible and as I've pointed out is in work
- but I do think we should develop a clear sense that it's
important to us that we run these things properly even if it means
direct local developer reproducibility is impacted.
Thanks! Monty
_______________________________________________ Mailing list:
https://launchpad.net/~openstack Post to :
openstack@xxxxxxxxxxxxxxxxxxx Unsubscribe :
https://launchpad.net/~openstack More help :
https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
