launchpad-dev team mailing list archive

[Ian Booth <ian.booth@xxxxxxxxxxxxx>]

Hi

I've been conflicted about an aspect of our current QA process. Well
there's several things that haven't been sitting quite right with me and
I've since been pointed to bug 297614 which highlights several of the
issues that exist. But there's one thing in particular which occurred
during the last rollout that I think for which we could consider a short
term fix, while the broader, longer term strategy matures.

As you will be aware, to deploy a particular revision, all previous
revisions generally have to be marked as "qa-ok" or "qa-untestable". So
far so good. However, a case arose where a revision was deployable in
that it did not break anything, but it still didn't necessarily fix the
issue described by the associated bug (turns out later it didn't). It
was suggested to me that the bug be marked "qa-ok" since a primary goal
is to test for deployability but I don't agree with this approach. For
starters, it disagrees with the bug state descriptions on the QA process
page:

<snip>
States of QA
    * qa-ok: The branch is verified and implements successfully what it
proposes.
    * qa-untestable: The branch cannot be tested separately of the rest
of the fix or cannot be tested at all, and is harmless.
    * qa-bad: The branch is verified and doesn't work as it should or
causes regressions.
    * qa-needstesting: The branch has landed and is waiting to be QAed.
<snip>

I've always in the past adopted a similar approach - marking a bug as
qa-ok/fixed when it is not is misrepresenting the true state of the bug
and has obvious downstream implications for process, deployment
readiness and also metrics gathering.

Having said that, I can see the point of the discussion I was having at
the time - we do not want to block a rollout unnecessarily. So I propose
the introduction of a new QA state which does not *require* us to
sometimes lie about the state of a bug - "qa-deployable". This has the
following characteristics:

- bug displayed as yellow in the deployment report (cf red or green)
- not a deployment blocker for subsequent revisions
- bug is still considered unfixed so will appear in any "what's
currently broken" type reports
- subsequent branches to correct the qa failure will use fixes=xxxxx
against the original bug number (rather than any new one that may be
raised to capture the QA defect) ensuring easier traceability for work
done to solve a particular problem

The addition of this new tag is not meant to be a solution to any longer
team or broader process changes currently under consideration via the
TrackingQualityAssurance LEP etc. However, I see it as a quick win that
can help us now in the short term.

I've got my fire proof suit on, but in case people see value in the
suggestion, we can take it to the next step which is scoping out the
changes required to LP, the QA tagger, the deployment reports etc. I'm
not intimately familiar with the internals of every affected tool or
component, but I doubt it would be too much work to implement? Is this
something we should or want to consider?

Thoughts?