launchpad-dev team mailing list archive

[Karl Fogel <karl.fogel@xxxxxxxxxxxxx>]

I thought it might be interesting to show how we're measuring the
effectiveness of a relatively new feature, the +patches report [1].

This isn't nearly the final word on this feature, for several reasons:
1) it's only been deployed for a couple of months; 2) Jorge Castro is
planning to talk about it with Ubuntu maintainers at UDS in a couple of
weeks; and 3) the particular statistic I'm measuring here is not the
only possible one we could measure.  In fact, I'm posting here partly to
hear other people's thoughts on what data to gather.

So, with all that in mind...

The goal of the +patches report is to make it easier for packagers and
upstream developers to find patches that could be upstreamed, and to
enable them to evaluate those patches quickly.

How can we tell if it's succeeding?  Well, one way is to see if the
average time between when the last patch is attached to a bug and when
<some bugtask on that bug> is closed is going down.

That's not quite as simple as it sounds, because:

  * We need to compare equal periods of time on each side of the
    deployment of the feature -- otherwise, we're comparing two months
    of having the feature against years of not having it.  The feature
    would look wildly successful, because any patch->closing cycle that
    takes place entirely after the deployment of the feature would (by
    definition) finish in under two months, while the cycles from before
    then can last arbitrary lengths of time.  (As you may have guessed,
    I didn't think of this at first, and was amazed at how effective the
    feature appeared... *appeared*... to be :-) ).

  * There's an overlap period: some patches appeared before the feature
    was available while their bugtasks were closed after the feature was
    available.  It's hard to guess how much the feature may have helped
    in those cases; I chose simply to eliminate such bugs from
    consideration, and look only at the "pure" cases on each side.

  * We only want to consider bugs belonging to teams/targets on which
    +patches reports have been done a lot.  The point of all this is to
    measure the effectiveness of the feature -- so rather than compare
    teams that use it with teams that don't, we want to compare teams
    that use it now with the same teams before they used it.

  * Various other interesting considerations.  See the code [2] if
    you're curious.

As of today, the results show the feature having no effect _in the
chosen measurement_:

  Average time to close a patch, before +patches: 
    10.17 days (considering 85 closings)

  Average time to close a patch, after +patches: 
    10.65 days (considering 196 closings)

This is not a huge surprise -- the feature is new, and only a few teams
are using it (I can see that from the access stats).  We could also ask:
is the feature enabling *more* bugs-with-patches to be closed than would
otherwise be closed?  In other words, maybe they take about the same
amount of time to get handled, but more of them get handled.

(You might think this is indicated by the "considering NNN closings"
clauses above, which shows more than twice as many closures happening
since the feature's deployment.  However, I think one would need to dig
in a bit more before concluding that.  There are many other things that
affect overall bug activity)

This is a pushbutton query: we can run it at any time, as it's based on
historical data that Launchpad preserves.  So we're going to run it
again in a couple of months, and meanwhile I'm going to talk with Jorge
Castro after UDS to see how teams think they're using the feature, and
how they think it's benefitting them (if at all).  We may change some of
what we measure based on that feedback.

So, the more interesting mail comes later.  But I wanted to show you
this now, in case anyone has deep thoughts on feature measurement.

-Karl

[1] http://blog.launchpad.net/bug-tracking/getting-patches-upstream
[2] https://code.launchpad.net/~kfogel/+junk/patches-view-stats