ubuntu-bugcontrol team mailing list archive

[Gema Gomez <gema.gomez-solano@xxxxxxxxxxxxx>]

Dear all,

I am preparing a blueprint for UDS and would like to gather your
experience and minds to help me get it right, based on your experience
regarding defect control. I am a member of the QA Team.

We are trying to improve our testing and are going to need help
prioritizing the areas that are in worse shape in terms of test
coverage. In the past, I have always started such task assessing the
quality of a system by looking at the defect database and trying to do
some analysis that helped me understand where we are and where most of
the problems live. We'll also need to measure code coverage, but that is
a conversation to be had outside of this mailing list.

In a nutshell, I need to come up with a classification strategy that
makes sense to be able to look at clusters of defects and figure out why
they are there and if QA can do anything to improve on those areas.

The usual questions to be answered for each defect would be: was this
found by a test case? Is the test case automated? if not, can it be
automated? If there wasn't a test case, could we add one for it?
automated? In which phase of the release cycle was it found? In which
phase of the release cycle could have been found? Is it worth automating
(i.e. not a marginal case or strange HW set up)? etc.

This is the blueprint as it stands:
https://blueprints.launchpad.net/ubuntu/+spec/qa-metrics

In the test escape analysis and defect analysis front, I have this so far:

   * Test escape analysis and defect analysis
        # of bugs that escaped our testing (and of these, how many
should have been found by testing, automated or manual/adhoc?)
        # of bugs found per month during our testing activities
    # of regressions found by our test suites: this is meant to measure
the test cases effectiveness, we could keep stats of defects found /
test case, to know which test cases are the most effective at finding
defects and be able to write more of those.


This kind of analysis/classification tends to be mostly manual at the
beginning and then going forward we can add tags as we deem appropriate
if we want to continue tracking any of the classes... we would like to
come up with a way of automating it as much as we can so that we can
concentrate on the test-facing bits, which are extracting the
information from the raw data. We have probably tags on the bugs that
could help with the classification.

So, based on your experience, how would you guys go about this task, do
we have enough information in launchpad already or do we need to tag
defects in a different way to allow us to do that? I don't mind reading
through a long list of defects and classifying them but I am so lost
that I wouldn't even know where to start extracting defects or which
list would be optimal set of defects to get started.

This information is instrumental to get our testing right on track, so
any help, tips, new ideas about questions to be add, would be much
appreciated. Please, copy me on your replies, as I am not a member of
bug control and therefore I am not subscribed to the list.

Best Regards,
Gema