launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/21/2011 10:16 PM, Francis J. Lacoste wrote:
> (Posted on the blog at http://blog.launchpad.net/general/5-9-23-51-and-other-
> numbers)
> 
> We are now two months away from our next Thunderdome. How are we doing in 
> regards with the objectives set for that milestone? You may recall from my 
> last post the objectives:
> 
>    * have no timeouts with a cut-off at 9s;
>    * have an empty critical bugs queue;
>    * getting a slot free on our ‘Next’ queue.
> 
> We practically achieved the first objective! Today, we lowered the hard 
> timeout to 9s and this didn’t increase our number of daily timeouts. We don’t 
> have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But 
> we get on average 650 requests timing out in a day. That’s less than 0.0001% 
> of our traffic.

650 / 8M = 0.008% (0.00008 as a fraction). But still, very well done.

> 
> These remaining timeout bugs are part of our second objective. On that front, 
> we are in a more difficult position. We have 259 critical bugs to close. That 
> went up since last time! What went wrong? Well, we had less people working on 
> critical bugs for once. That’s been fixed this week when the Orange squad 
> rotated back on maintenance. We again have two full squads working on critical 
> bugs. Second, we modified our OOPS reporting to show all timeouts happening, 
> not only the ones occurring the most often. That resulted in about 30 new 
> timeouts filed. (See the hight red bar at the start of the graph). Fortunately 
> for us, the rate of new critical bugs is declining.  We are at about 23 on 
> average in the last two weeks. That’s still high and some of those are related 
> to JS regressions escaping to production because our Windmill test 
> infrastructure is disabled. This means that 51 is now the magic number. We 
> need to close 51 of these critical bugs per week to reach 0 by the 
> Thunderdome. That was the number we closed in our best week, just before the 
> number of people working on criticals was reduced. So we’ll also need to 
> reduce the number of new critical bugs found each week to succeed here.

I realize 259 is a lot, enough that it is hard to get a handle on. Have
you gone through them at all to see if there are bulk-fixes, things that
are already fixed, etc. I'm certainly guessing there is a fair bulk that
are going to be similar in effort, and a really long tail of ones that
are hard to handle (like problematic timeout pages, etc.) It would be
interesting to get a feeling for where the knee of the curve is, though.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2xZJEACgkQJdeBCYSNAAPnjQCgjaFCsMlIP93mbcyD+lOffBOZ
GYIAnjmJKTqeaQRNvTCdL+lR52dhy1M0
=3epM
-----END PGP SIGNATURE-----