maas-devel team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

It seems to me that we have a bit of a testing dichotomy and I can see 
advantages and disadvantages to both approaches.

Let me see if I can boil it down to these two things:

 1. patch and mock every test that is likely to call RPC. Gavin says this can 
be mitigated by writing code differently (but doesn't say how :)), but this 
won't help with the hundreds of existing tests of course.

 2. Have a permanent mock cluster that returns canned results as set for each 
test. This could end up masking real bugs by using results that don't match 
reality.

If we take approach #1, then we'll get some developer WTFs when writing new 
tests that fail because some underlying code is trying to do RPC.

If we take approach #2, then we'll get developer WTFs when writing new tests 
that that fail because some underlying code is trying to do RPC.

;)

The reason it worked more smoothly with the Celery fixture was because it was 
one-way traffic; that is we invoked a task and didn't expect any results.  We 
just needed to patch out anything that failed in an irrelevant way.  Gavin is 
right here in that it was testing way more stuff than we really needed to as 
it was kicking off Celery tasks (synchronously) for a huge number of tests.

I think we can have some middle ground here.  If we make the RPC fixture 
available by default (as in #2, like Celery was) but make it return sane 
errors when some RPC gets called that is not mocked yet, then we get notified 
that you're in one of these situations:
 1. you might not be unit testing what you think you are
 2. you might need to set a canned response

We would need a similar thing in reverse for the pserv tests, so that the test 
case has a default region fixture.

Thoughts?

J