vm team mailing list archive

[Tim Cross <tcross@xxxxxxxxxxxxxxx>]

From: Uday S Reddy <u.s.reddy@xxxxxxxxxxxxx>
Subject: Re: [Vm] compilation warnings
Date: Sun, 4 Jul 2010 11:01:23 +0100

> Tim Cross writes:
> 
>> > You are probably misunderstanding me.  My question is basically
>> > whether your testing has covered (i.e., executed) all the code that
>> > has changed.  Moreover, in the unibyte/multibyte case, we need to test
>> > it with multibyte text.  
>> >
>> 
>> My comment was to your reference to tools to manage test coverage. It would be
>> very useful to have a 'standard' collection of test messages that we could
>> use. 
> 
> No, it seems that we are still talking at cross-purposes.
>

My fault for not being clear. 

A test coverage tool would be of some use, though persoanlly, I've never found
them worth the additional overhead they require to maintain. This is likely
very dependent on the tools being used. I've mainly used them with Java and
have used one for C back in the early 90s while working at the Distributed AI
Labs. It did help in catching some common C bugs that are difficult to find
otherwise, but required considerable effort re: instrumenting the code. From
memory, it was a code coverage tool that came out of one of the DARPA research
initiatives that funded most of our work. In general, compared to good unit
testing or my preference, using test driven development, I've found them too
much work to maintain for the benefit they provide.

My point was a more general testing one. Given the existing code base and the
overhead necessary to retro fit full unit tests, a regression test suite would
probably provide the most return for effort. I think we should use a variety
of different testing strategies. What I'm trying to do is think about how we
can start moving towards something that will increase the quality of our
results while not becoming so much of a hassle that we avoid it because of
time pressure etc. 

> In the absence of such a tool, what I do is to use the Emacs
> debuggers.  Set break points around the code I want to test, and trace
> through the code step by step to examine whether everything is being
> covered.  Also, remembering the Dijkstra adage, "testing can only show
> the presence of bugs, not their absence" I try hard to imagine what
> else could happen which is not represented by my test cases.
>

While using the debugger and step testing is better than nothing, it only
provides a very limited form of testing and is very time consumeing. Have you
tried using testcover.el? That would probably be faster and would give a
similar result. At the very least, it would probably assist in identifying
test cases. 
 
> For the recent Thunderbird compatibility code, I wasn't able to do
> this because I didn't have enough Thunderbird email to test it with.
> As you saw, problems remained when it got published.
> 

Yes, this is an issue. What is the criteria we use to assess adding features
like this? I've always thought it a potential source of maintenance overhead
to add compatibility with other mailbox formats. Is there a plan or blueprint
for what mailbox formats we would like VM to support?

>> We could possibly then also develop a standard set of regression tests.
>> My point is that it will take a lot of work to develop something like this.
>> However, this does not mean it isn't worthwhile. In fact, given the code base
>> we have, it could be argued it is essential. It would be great if we could
>> just run a script after a build that executed a set of tests - while this
>> would not be sufficient in itself, it would add to our confidence that changes
>> don't introduce new problems.
> 
> This would be system testing.  To me, system testing doesn't give the
> same kind of confidence as unit testing.

Currently we have no systematic testing at all, so anything would be
an improvement. My preference is towards unit testing and test driven
development in particular. However, adding unit tests to an existing code base
is a very difficult and slow task. 

What I'm thinking about is regression testing where we would have a suite of
tests that could be applied automatically after a build. The usual way I've
used this in the past is to have scripts that do a checkout of the code each
night, build it and apply the test suite. Generally, if any of the tests fail,
they would be dealt with as a matter of priority the next morning. While this
gives no guarantee of code coverage, it does provide increased confidence.
Especially if we define tests to identify bugs prior to fixinig them and then
add those tests to the regression suite. To some extent, it is a pity we are
bazaar based rather than git based as there is a good code review tool called
gerrit that is quite nice and a good way to manage peer review of code
contributions (its what Google and the Android project use). I think peer
review is an extremely important aspect of development, especially in
distributed projects. Gerrit takes a lot of the pain out of doing this.

> 
> It would be useful to start collecting email messages that could form
> a good test suite.  It is a bit hard to do because email messages
> can't be shared easily.  But we should try.

I find my spam folder to be quite useful in this respect. We can share them
with no issues and they are often good examples of badly formatted messages. 

> 
>> Yes. I'm trying to deal with them in 'groups' of similar messages and then
>> commit each group so that it is easier to extract specific bunches of fixes. I
>> am also using my compiler-fixes branch as my current VM version as an
>> additional ad hoc testing process. 
> 
> That is good.  Can you make sure that you commit each group
> separately?  That would make it easier when we merge them with the
> trunk.

Sorry, but isn't that what I said I was doing or do you mean pushing them as a
goup up to my launchpad branch?

Tim