← Back to team overview

launchpad-dev team mailing list archive

Re: public xmlrpc requests looking bad after last rollout

 

On 14/07/10 12:12, Michael Hudson wrote:
On 14/07/10 11:42, Gary Poster wrote:

On Jul 10, 2010, at 6:57 PM, Gary Poster wrote:

The average request time for of the public xmlrpc server (that is,
requests that match
^https?://xmlrpc\.) just before the last rollout was .21 seconds (see
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-06_2010-07-07/categories.html).


The past two days, the average request time for these requests was
over 8 seconds
(https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-08_2010-07-09/categories.html
and
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-09_2010-07-10/categories.html
; click on "Mean time" to sort it to the top).

(1) The regex identifying these as public xmlrpc requests was
incorrect. We'll fix it.
(2) From
https://devpad.canonical.com/~stub/ppr/lpnet/daily_2010-07-13_2010-07-14/top200.html
(these are the top 200 hits, fwiw), it looks like these two URLs are
the culprits:

https://xmlrpc.lp.internal:8097/codeimportscheduler/getJobForMachine
(avg time 8.29 requests * 1066 requests)
https://xmlrpc.lp.internal/codeimportscheduler/finishJobID (avg time
7.27 requests * 428 requests)

Oh. So the thread title is wrong :-)

Neither of these methods really does very much, although finishJob is a
little involved. Can we get a soft-oops type break-down of the queries
they execute?

Ah, some of them are triggering soft timeouts, so that we can see stuff like https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1655XMLP1

UPDATE CodeImportMachine SET heartbeat=CURRENT_TIMESTAMP AT TIME ZONE 'UTC' WHERE CodeImportMachine.id = %s took 8 seconds in this case. I think this must be a result of some kind of locking, but I can't really imagine what as this is the only method I can think of that writes to this table routine. Maybe many concurrent calls to the same method are contending with each other? Seems a tad unlikely though.

Cheers,
mwh



Follow ups

References