launchpad-dev team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx>]

(Posted on the blog at http://blog.launchpad.net/general/5-9-23-51-and-other-
numbers)

We are now two months away from our next Thunderdome. How are we doing in 
regards with the objectives set for that milestone? You may recall from my 
last post the objectives:

   * have no timeouts with a cut-off at 9s;
   * have an empty critical bugs queue;
   * getting a slot free on our ‘Next’ queue.

We practically achieved the first objective! Today, we lowered the hard 
timeout to 9s and this didn’t increase our number of daily timeouts. We don’t 
have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But 
we get on average 650 requests timing out in a day. That’s less than 0.0001% 
of our traffic.

These remaining timeout bugs are part of our second objective. On that front, 
we are in a more difficult position. We have 259 critical bugs to close. That 
went up since last time! What went wrong? Well, we had less people working on 
critical bugs for once. That’s been fixed this week when the Orange squad 
rotated back on maintenance. We again have two full squads working on critical 
bugs. Second, we modified our OOPS reporting to show all timeouts happening, 
not only the ones occurring the most often. That resulted in about 30 new 
timeouts filed. (See the hight red bar at the start of the graph). Fortunately 
for us, the rate of new critical bugs is declining.  We are at about 23 on 
average in the last two weeks. That’s still high and some of those are related 
to JS regressions escaping to production because our Windmill test 
infrastructure is disabled. This means that 51 is now the magic number. We 
need to close 51 of these critical bugs per week to reach 0 by the 
Thunderdome. That was the number we closed in our best week, just before the 
number of people working on criticals was reduced. So we’ll also need to 
reduce the number of new critical bugs found each week to succeed here.

Finally, on our last objective, like I already mentioned, the Orange squad 
switched back to maintenance. That’s because they completed the final bits of 
the project they were working on: Sharing translations between upstreams and 
Ubuntu. (Expect a blog post about it next week once everything is available to 
the general public.) That means that we need to complete 5 more projects to 
free a slot in our Next queue. That will be a stretch, but not impossible 
since of the two currently in progress, one should be completed in a couple of 
weeks and the other before the end of next month.  And two of the remaining 
ones are much simpler than what we have worked on until now.

I’ll post again next month to see how the final stretch looks like.

-- 
Francis J. Lacoste
francis.lacoste@xxxxxxxxxxxxx

Attachment: signature.asc
Description: This is a digitally signed message part.