openstack team mailing list archive

[Brian Schott <bfschott@xxxxxxxxx>]

One thing we saw in the list and experienced first hand is that your Hudson server uses a pre-configured environment and ours pulls virtual env every time.  We had failures on trunk that yours missed because of new pip pulled versions.

A fresh run_tests.sh -f needs to work.  It is the only guaranteed sanity test everybody can replicate.  It might pull upstream bugs, but better to be ahead of that curve than behind.

We are now triggering on trunk commit emails and running Hudson on both trunk and our hpc-trunk.  We can forward to a list or autogenerate bug report somehow.  

Suggestions:
1. Let other teams setup their own trunk Hudson servers with preferred configs (os, virt, db, net) and doc how to forward results up (new list?). This way you quickly find commits that break mysql or Xen before you forgot what you did.
2. Jay already listed my other suggestions...

Sent from my iPhone

On Feb 16, 2011, at 11:44 PM, Paul Voccio <paul.voccio@xxxxxxxxxxxxx> wrote:

> Jay,
> 
> Thanks for throwing this out. How would we build this with Hudson? What
> would a "standard deploy" of Nova even look like for integration tests?
> We've also bounced the idea within our team of not allowing code commits
> if the code to test ratio decreases but I'm not sure if that would work
> for such a large project like this one.
> 
> pvo
> 
> On 2/16/11 4:27 PM, "Jay Pipes" <jaypipes@xxxxxxxxx> wrote:
> 
>> Hey all,
>> 
>> It's come to my attention that a number of folks are not happy that
>> Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :)
>> 
>> First, before going into some suggestions on keeping trunk more
>> stable, I'd like to point out that trunk is, by nature, an actively
>> developed source tree. Nobody should have an expectation that they can
>> simply bzr branch lp:nova and everything will magically work with a)
>> their existing installations of software packages, b) whatever code
>> commits they have made locally, or c) whatever specific
>> hypervisor/volume/network environment that they test their local code
>> with. The trunk branch is, after all, in active development.
>> 
>> That said, there's *no* reason we can't *improve* the relative
>> stability of the trunk branch to make life less stressful for
>> contributors.  Here are a few suggestions on how to keep trunk a bit
>> more stable for those developers who actively develop from trunk.
>> 
>> 1) Participate fully in code reviews. If you suspect a proposed branch
>> merge will "mess everything up for you", then you should notify
>> reviewers and developers about your concerns. Be proactive.
>> 
>> 2) If you pull trunk and something breaks, don't just complain about
>> it. Log a bug immediately and talk to the reviewers/approvers of the
>> patch that broke your environment. Be constructive in your criticism,
>> and be clear about why the patch should have been more thoroughly or
>> carefully reviewed. If you don't, we're bound to repeat mistakes.
>> 
>> 3) Help us to write functional and integration tests. It's become
>> increasingly clear from the frequency of breakages in trunk (and other
>> branches) that our unit tests are nowhere near sufficient to catch a
>> large portion of bugs. This is to be expected. Our unit tests use
>> mocks and stubs for virtually everything, and they only really test
>> code interfaces, and they don't even test that very well. We're
>> working on adding functional tests to Hudson that will run, as the
>> unit test do, before any merge into trunk, with any failure resulting
>> in a failed merge. However, we need your help to create functional
>> tests and integration tests (tests that various *real* components work
>> together properly).  We also need help writing test cases that ensure
>> software library dependencies and other packaging issues are handled
>> properly and don't break with minor patches.
>> 
>> 4) If you have a specific environment/setup that you use (Rackers and
>> Anso guys, here...), then we need your assistance to set up test
>> clusters that will pull trunk into a wiped test environment and ensure
>> that a series of realistic calls to the Nova API are properly handled.
>> I know some of you are working on getting hardware ready. We need help
>> from the software teams to ensure that these environments are
>> initialized with the exact setups you use.
> 
> Still working on this. We're hoping to have something built out in the
> next couple of weeks. Man, someone should write some hardware emulation
> stuff so I don't have to wait on real gear. ;)
> 
>> 
>> The more testing we fire off against each potential merge into trunk,
>> and the more those tests are hitting real-life deployment
>> environments, the more stable trunk will become and the easier your
>> life as a contributor will be.
>> 
>> Thanks in advance for your assistance, and please don't hesitate to
>> expand on any more suggestions you might have to stabilize trunk.
>> 
>> -jay
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is prohibited.
> If you receive this transmission in error, please notify us immediately by e-mail
> at abuse@xxxxxxxxxxxxx, and delete the original message.
> Your cooperation is appreciated.
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp