launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Tue, Apr 19, 2011 at 9:43 PM, John Arbash Meinel
<john@xxxxxxxxxxxxxxxxx> wrote:
> This is really great news. I'm curious how this works with your in-DC
> queuing? At least, from what I remember in the past, your stats did not
> take that into account.

Wasn't measured in the first set of stats, still not in the second.
There is a High priority bug in stub's queue to get some sort of
approximate data here. We know from coarse measurements that we're in
much better shape than we were. I fully expect that when we get
concrete data we'll find more issues to fix.

> It is really nice to see that both the high-end (timeout reduced to 10s)
> and the low end (99th percentile at 1.59s) is getting better. It will
> certainly be interesting to see if/when you switch from "how can we make
> this terrible query less than 10s" to "how can we make these fast 2s
> queries faster (<1s)".

We're already there :). Primarily because some pages need to run e.g.
N (say 3) similar queries and they all need to be <timeout/N seconds
to work at all. Sadly we have some things that are still firmly in the
10-second query window; so I think its going to be a bit of both for
quite a while.

>>
>> We're down to 316 timeouts - 0.004% of requests, so, with an exception
>> for Question:+index (which could spike by 200 timeouts a day) the hard
>> timeout has been dropped to 10 seconds.
>>
>> If it looks good - and it may - we'll drop to 9 seconds later this
>> week; meeting Francis challenge for the Epic and leaving only 4
>> seconds to go to reach my long term timeout of 5 seconds.
>>
>> Of course, we're going to need better and better code and schemas to
>> continue improving things, but we're in pretty good shape!
>>
>> -Rob
>
> Do we know how many exceptions we have at 10s, and how many exceptions
> we would need at 5s?

We have 2 in place at the moment (10s). I haven't run the data to
predict how many we'd need if we jumped straight to 5seconds. I
suspect about 30 based on the total timeouts we see.

-Rob