launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

In summary, I propose that:
- - we tag revisions, not bugs, for deployment QA
- - deployment reports should have a (separate) list of revisions that
  need QA, with links to their merge proposals.
- - deployment reports should use a stricter definition of "deployable".
- - it would be nice to have a list of why revisions are not deployable.
- - rollbacks should be tagged in a way that indicates the affected range.
- - rollbacks should not, themselves, require QA.

I've had deeper experience of the QA process this cycle, because I'm
release manager, and because I wrote a script when the qa-tagger was broken.

We combine bugfix verification with deployability assessment, but I
think it might make sense to separate them.  A change which does not
introduce regressions is safe to deploy, even if it does not fix bug it
was intended to.  It does happen, e.g. bug #671665.

Sometimes branches are reused to fix new bugs.  This should never change
the status of a revision that has already been QAed, but it does.

In order to fix these issues, I think we should update our data model so
that revisions can be tagged.  They can then be tagged qa-needstesting,
qa-ok, etc.  A qa-ok

A release manager needs to chase down QA.  It would be useful if the QA
report listed which revisions need QA and who should do it.  We have
some of this information, but it's intermixed with other information,
and often greyed-out.  A simple list of revisions needing QA or needing
fixes, plus a link to the relevant merge proposal(s) would be better,
and we already have all the necessary information.  (You can now find
merge proposals by revno).

A release manager needs to pick a revision to deploy.  The current
reports don't directly indicate which revisions are safe to deploy *as
tip*.  Instead, they may say "X is deployable if Y is deployed".  To a
release manager, this means that X and all revisions up to Y are not
deployable as tip.  Y may be deployable as tip, but there may be other
reasons why Y is not deployable.  A list of which revisions are
deployable *as tip* would be excellent.

Secondarily, a list of the reasons why revisions are not deployable
would also be helpful.  For example:
r3-12 are not deployable (issue in r3 fixed in r13)
r10+ are not deployable because r10 needs QA
r14+ are not deployable because r14 is qa-bad.

The way we mark rollbacks is confusing.  It would be clearer to indicate
what range of revisions was included in the rollback.  If we support
tagging revisions, then we could tag the new revision
"rollback-$OLDREVNO" or "fix-$OLDREVNO".  If we continue tagging bugs,
then "rollback-$OLDREVNO-$NEWREVO".  This way, it's clear what range of
revisions is broken.

If X is a revision, and revision Y is a rollback of revision X, revision
Y can cause a regression.  Since any revision following X might
accidentally depend on it, thorough QA of Y would require repeating the
QA for all revisions between X and Y.  This is not practical.  X might
have been rolled back because it lacked QA, so requiring QA of Y seems
unfair, too.  Therefore, I propose that QA should not be mandatory for
rollbacks.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkz+rH8ACgkQ0F+nu1YWqI1lPgCfQo10drXa8+PasvpBIrHTyR6v
9boAn1T1AsYjneOYBFKooIrYqgZTlpq7
=2hvM
-----END PGP SIGNATURE-----