launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/3/2011 3:29 AM, Robert Collins wrote:
> On Wed, Mar 2, 2011 at 3:06 PM, Robert Collins
> <robertc@xxxxxxxxxxxxxxxxx> wrote:
>> We're continuing to make steady progress on timeouts - there are 5
>> timeout improve/fix patches in the deploy queue at the moment.
> ...> I'm going to ask the losas to drop the request timeout another second
>> now : this may cause some more pages that are on the edge to start
>> failing on their first hit (or second if its not a cold cache
>> situation). if the spike is large, we can roll it back at a moments
>> notice.
> 
> Its been 24 hours now, and the report from yesterday shows no increase
> in timeouts: we've shaved another second of our backstop without
> increasing our failure rate at all. (Though the daily reports show can
> show variation due to use activity, a large spike would have been
> visible.).
> 
> We have made significant traction on BugTask:+index, the worst
> offender for a while now:
> 166 /  296  BugTask:+index (1st Mar)
> 123 /  188  BugTask:+index (2nd Mar)
> 
> I intend to leave the timeout where it is for another month as we
> continue to fix things; if we get below 0.005% requests timing out,
> I'll consider another reduction mid March.
> 
> Cheers,
> Rob
> 

Seems like something you could push to failure, and then back off, if
you really wanted to push it. 1s increments per day/week, then backoff
when it starts increasing significantly, and then focus on performance
of remaining timeouts. I just have a strong suspicion that 99.9% of your
responses are way under the timeout, so you have a huge gap between
where the threshold is, and where it has an actual effect. (The requests
that fail today would be taking 20+s, the requests that succeed are down
in the <10s range. Notching from 15 to 14 certainly doesn't seem like it
would do much, which is certainly what you saw.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1vW4QACgkQJdeBCYSNAANfbQCgo9tOq3kr4Si9s7JkF6QJeFuO
zrUAoKF3NZM88g+8+Pe98CJcDb/HSX1U
=SEsy
-----END PGP SIGNATURE-----