launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Sun, Nov 14, 2010 at 12:00 AM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> We had a period - about 36 hours long - where the xmlrpc internal
> server was simply saturated and overloaded. During this period folk
> would experience a backtrace as described in
> https://bugs.launchpad.net/launchpad/+bug/674416.

There were no reports that I can find of this symptom overnight - I
looked at new Launchpad bugs, lp-ops, #bzr, #launchpad and
#launchpad-dev. I think we're clean. \o/

Colin Watson has noted that we have some cleanup to do - we need to
trigger a 'branchChanged' event for all the branches which had that
event crash.

> Time Out Counts by Page ID
>
> Hard    Soft    Page ID
> 384     578     CodehostingApplication:CodehostingAPI
> 94      832     CodeImportSchedulerApplication:CodeImportSchedulerAPI
> 89      14      Person:+commentedbugs
> 14      39      BugTask:+index
> 10      16      Archive:EntryResource:getBuildSummariesForSourceIds
> 4       0       https://api.edge.launchpad.net
> 4       0       ProjectGroup:+milestones
> 2       37      Distribution:+bugtarget-portlet-bugfilters-stats
> 2       7       DistroSeriesLanguage:+index
> 2       4       Distribution:+archivemirrors
> (from https://devpad.canonical.com/~lpqateam/lpnet-oops.html#time-outs)
>
> Again, if it was over-saturation causing the issue, we should expect
> the first two rows to stay constant (or nearly so) over the remainder
> of the day. Its been stable for a good 30+ minutes now - in fact
> +commented bugs just passed the codeimportschedulerAPI, with the xml
> resources staying constant. This is a good sign.

Current figures are:

Time Out Counts by Page ID

Hard	Soft	Page ID
384	578	CodehostingApplication:CodehostingAPI
173	36	Person:+commentedbugs
94	832	CodeImportSchedulerApplication:CodeImportSchedulerAPI
20	107	BugTask:+index
11	1	DistributionSourcePackage:+addquestion
10	18	Archive:EntryResource:getBuildSummariesForSourceIds
5	2	ProjectGroup:+milestones
4	153	Distribution:+bugtarget-portlet-bugfilters-stats
4	32	MaloneApplication:+bugs
4	0	https://api.edge.launchpad.net

So the two xmlrpc calls have not failed over the last 7 hours. Excellent.

-Rob