openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

Hi Gavin! Comments inline...

On Mon, Jan 16, 2012 at 10:18 AM, Brebner, Gavin <gavin.brebner@xxxxxx> wrote:
> I’m interested in running some “white-box” tests that check scalability and
> limits of parts of a Nova system.  I want to start from a
> full working configuration however, as I get that from my dev team and it
> includes all sorts of settings that would be tedious
> and error prone to reproduce in other ways. My thought was that good use
> could be made of some of the “fake” drivers
> that are present – I’m hoping to be able to make some minimal code/config
> changes to swap the driver on my host node(s)
> for a fake one, restart the services and run tests.

Yes, this is an idea I've been playing around with as well. In
performance testing, if you cannot isolate (to the best of your
ability) other components in a multi-component system, it is virtually
impossible to get trustworthy data. Everything boils down to the
scientific principle of "all other things being equal". In other
words, if I'm trying to produce regression test results for a
particular component (say, Glance) from one revision to another, I
cannot do so if the other components in the test system are changing
or the system load itself is in flux.

Fakes are useful when you are running repeated tests trying to
identify when a particular patch or revision either improves or
degrades performance of the system as a whole. By using fakes (which
either toss data to /dev/null or store persistent data in an in-memory
store), you can dramatically reduce the overall time to run a
particular test. By reducing the time taken for a test, you can test
multiple successive patches or revisions more easily, knowing that the
only thing you are changing is the implementation of a particular
component.

> Any caveats on this that I should be aware of ? For that matter has anyone
> tried this kind of thing before ? I’d be interested in hearing of your experiences if so.

There are a bunch of faked out drivers for various things in the core
projects. But there's not really any consistency to them, AFAICT.

What would be useful is to get a detailed description from you of the
exact questions you are trying to answer. That would enable us to
determine which fakes would be useful and/or need to be written in
order to isolate your environment enough to retrieve trustworthy data
from test results.

Cheers!
-jay