coapp-developers team mailing list archive

[Garrett Serack <garretts@xxxxxxxxxxxxx>]

[recycling back to the list ... make sure you use reply-all :) ]

For Category A, each tool has a fairly constrained output-for testing them individually, might we just as well setup an automatic run against a bunch of real-world projects?  I'm just not sure there is an automated way to validate that the results are what you wanted (although, one could argue that if you actually got out the packages you wanted, they are probably *what* you wanted.)

>> Testing is also as much about trying to break things as anything

Ah, yes. That is a good way to look at it. Hmmm. Actually, that goes a long way to thinking about testing; not so much the validation of what works, as an appropriate halt to bad inputs.

G

Garrett Serack | Open Source Software Developer | Microsoft Corporation
I don't make the software you use; I make the software you use better on Windows.

From: Ritchie Annand [mailto:wakarimasen@xxxxxxx]
Sent: Wednesday, April 21, 2010 9:00 AM
To: Garrett Serack
Subject: Re: [Coapp-developers] Thinking abotu testing CoApp's tools.

Is the bellwether of success a working app or library out the other side? There are probably test cases along those lines that could be done for A and D, like testbed apps that spit out something pretty simple when run. I imagine, though, something like that might suffice to a point, but you're contemplating UI apps as well which might not give up their issues from the command line.

Testing is also as much about trying to break things as anything. Putting together fake processes which *ought* to fail at some point along the tool chain can help make sure that the reported successes are not fake. Sign something wrong, put in a version of a library that should not pass the build rules, or throw in two side by side libraries that do slightly different things in the final test build and see whether you're actually getting the right version despite trying to confuse the build process with ambiguity.

-- Ritchie

----- Original Message -----
From: Garrett Serack <garretts@xxxxxxxxxxxxx>
Date: Wednesday, April 21, 2010 9:44 am
Subject: [Coapp-developers] Thinking abotu testing CoApp's tools.
To: "coapp-developers@xxxxxxxxxxxxxxxxxxx" <coapp-developers@xxxxxxxxxxxxxxxxxxx>

> I know this is early, but I'd like to get some ideas rolling
> around testing.
>
> We're going to be writing a few different classes of software
> for CoApp:
>
>
> (a)    A collection of command line
> tools/libraries (+msbuild plugins) to automate the tasks of
> adapting source code to Windows, and building packages. (Trace,
> ScanTool, mkSpec, mkProject, PGOMagic, mkPackage, SmartManifest)
>
> (b)   A client service that handles the nitty-gritty
> details of managing the dependencies, shared libraries, updates
> and consistency & validation (essentially all the brains under
> the UI)  (SharedLibraryConfig)
>
> (c)    A client GUI application that is the
> primary UI that end-users interact with (like synaptic)
>
> (d)   A command-line client for those of us with too
> many years at the keyboard & still like to script (like apt)
>
> (e)   A Visual Studio plugin for developers
>
>
>
> Ideally, we should have a plan for testing these-although items
> in category A don't lend themselves to easy testing.
>
> Category B is critical to be heavily tested and security
> audited, and we're going to have to go to town on that. Plus, it
> seems easy to me to understand how to write automated tests for it.
>
> Category C & D-and to a certain extent, Category E-are pretty
> lightweight and I think of them as wrappers around the services
> that Category B provides. I suppose these are testable, but to
> what point?
>
> I'd like to hear what ya'll think about strategies for handling
> testing of A/CDE.
>
> [Description: fearthecowboy]<http://fearthecowboy.com/>
>
> Garrett Serack | Microsoft's Open Source Software Developer |
> Microsoft Corporation
> Office:(425)706-
> 7939                                       email/messenger: garretts@xxxxxxxxxxxxx<mailto:garretts@xxxxxxxxxxxxx<mailto:garretts@xxxxxxxxxxxxx%3cmailto:garretts@xxxxxxxxxxxxx>>
> blog:
> http://fearthecowboy.com<http://fearthecowboy.com/<http://fearthecowboy.com%3chttp:/fearthecowboy.com/>>                                      twitter: @fearthecowboy<http://twitter.com/fearthecowboy>
>
> I don't make the software you use; I make the software you use
> better on Windows.
>
>
>
>
>
>
>