launchpad-dev team mailing list archive

[Ian Booth <ian.booth@xxxxxxxxxxxxx>]

Hi Julian

> 
> As already noted, we really do need to move to a QA per-revision system 
> longer-term.
> 
> As for qa-deployable, can you summarise the problems with using qa-ok?  I 
> didn't see any hard issues in your email but then I might have been reading it 
> wrong.  You left me tantalisingly close to understanding why when you said:
>
>> marking a bug as
>> qa-ok/fixed when it is not is misrepresenting the true state of the bug
>> and has obvious downstream implications for process, deployment
>> readiness and also metrics gathering."
> 
> but it would be great to enumerate these implications!
>

My reply to Aaron's recent email sums it up a bit. It's the fact that we
seem to be confusing the meaning of the tag from a release management
perspective vs a QA (verification) perspective. As a release manager you
want to know if a revision is deployable. From a QA perspective, you
want to know that the revision is verified as solving the defect
identified by the bug and hasn't caused any regressions. These are
overlapping but different concerns. Given the expected usage of the qa
tags seems to be to indicate deployability rather than the QA meaning
(for want of a better term), my contention is:
- the tags are misnamed: "qa-ok" has a certain implied meaning
- they are misdefined to match the implied QA meaning: eg the
definitions of qa-ok and qa-bad on QAProcessContinuousRollouts talks
about the QA aspect

So if a revision is marked as qa-ok because it can be deployed, but
doesn't actually fix the problem is was designed to fix, and given the
*current* written definition of what qa-ok means on
QAProcessContinuousRollouts, then we would be misrepresenting that the
bug is fixed when it's not and so metrics gathering around number of
open defects, regressions etc could/would be compromised depending on
how such metrics were gathered. It could cause confusion also if someone
ran a report "+bugs?field.tag=qa-ok" expecting to see bugs which were fixed.

Read in conjunction with Aaron's and my discussion on this thread, I
hope that makes more sense now :-)