openstack team mailing list archive

[Soren Hansen <soren@xxxxxxxxxxx>]

2011/11/21 Duncan McGreggor <duncan@xxxxxxxxxxxxx>:
> Soren, I'm delighted to see this email come across the list -- thanks
> for jump-starting such a discussion. It probably goes without saying
> that I completely agree with your assessments above :-)

Great! :)

> The one thing that has bothered me most about the OpenStack unit tests
> has been that many of them are not really unit tests; the presence of
> functional or integration testing precludes that possibility.

Very true. Do you think we should spend some time splitting our tests
into "true unit tests" and all the rest? That way, we can easily choose
to only run unit tests or the whole thing.

> To counter up-front any objections to being too much of a purist or
> drinking too much of the TDD koolaid, let me say that I do believe in
> testing against real databases, across real networks, etc., but those
> should be tests that can be run optionally and explicitly stated as
> being integration tests or functional tests.

I think it depends on the type of dependency. Some things should
definitely not be involved when running unit tests (sqlalchemy+sqlite
for instance), but other things are perfectly fine IMO. E.g. a function
that validates whether something is a valid IP address may have started
its life as a homegrown routine, but eventually gotten replaced by a
routine that uses python-ipy or python-netaddr. That means the test
suite has a hard dependency on python-ipy or python-netaddr, but I still
think this falls squarely in unit testing territory.

> By using mocked objects (e.g., Michael Foord's mock library), unit
> test speeds will go WAY up.

We use python-mox quite a bit and have half a bajillion ad-hoc mock
implementations of various things.

I'm actually on a mission to replace many of these with more thorough
fake objects. I have a fake db implementation I've been working on for a
couple of days that gets me half way through the test suite in less than
a minute. It's supposed to eventually replace the countless ad-hoc mock
db implementations scattered across our test suite. They were great when
they were added, but IMO they're much more a liability now, because
they're impossible to maintain.

> If we could separate out everything that currently hits other systems
> in the OpenStack amalgam so that they use a separate runner
> (integration_tests.sh, functional_tests.sh, whatever....), and just
> keep the unit tests in the main test runner, not only will things run
> much more rapidly, but it will be a no-brainer for folks to start
> running them compulsively (before submitting patches, before
> submitting bugs, before asking for help on IRC, etc.).

Sounds like a great idea.

> As for the duplicate code, it's probably fairly obvious to everyone
> that we could build up a subpackage for testing, that pulls together
> all the commonly used code and simply import as necessary (as base
> classes -- OOP -- or instantiate as attribtues -- AOP).

That would be fantastic.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/