ubuntu-phone team mailing list archive

[Tony Espy <espy@xxxxxxxxxxxxx>]

On 09/04/2013 05:49 AM, Evan Dandrea wrote:
> Hi folks,
> 
> In another discussion, James Hunt raised the possibility of
> periodically checking for runaway processes on Touch, killing those
> consuming 100% CPU while creating a report to be sent to
> https://errors.ubuntu.com.
> 
> I've summarised the key points of that discussion here into a
> proposal. The hope of this is that it gives everyone a chance to
> provide input.

Is this a proposal for 13.10?

I understand the basic reasoning, but wantonly killing system processes
and then hoping that the system will always gracefully recover sounds a
bit risky to me.

Many system service have complex start-up sequences, and although they
*should* handle restarts properly, they may not, potentially leaving the
device in an inconsistent state.  Processes handled by upstart should
work, but what about helper processes ( eg. dhclient ), are all
guaranteed to be automatically re-started?

I'd argue that an OEM would want to test as many possible "kill
scenarios" as possible before ever releasing a device that implemented
such a mechanism.

Was any thought given to collecting the requisite information and then
rebooting?  Perhaps this is an optional behavior that could be configured?

Also as mentioned, this might be something an OEM might want to disable,
so it should be possible to easily defeat this on a per-device basis via
configuration file.

> == Examples ==
> 
> There are a few examples of this problem biting us already.

Two of the three bugs listed below "had" bitten us, but have been fixed.

> The original bug James ran into was:
> https://bugs.launchpad.net/ubuntu/+source/bluetooth-touch/+bug/1217865
> 
> Martin Pitt also raised one where two rogue system service processes
> constantly used 150% CPU (i. e. 1.5 cores):
> https://launchpad.net/bugs/1188404

This actually was one "rogue" process ( ofonod ), caused by an
implementation bug ( infinite loop ) which has been fixed.

> A few weeks ago there was a nasty timing bug which caused ueventd to
> use 100% CPU:
> https://bugs.launchpad.net/touch-preview-images/+bug/1190792

Also fixed by a change to ueventd ( yet another infinite loop due the
original design not requiring co-existence with udev ) last week.

> Whoopsie also had a memory corruption bug which caused 100% CPU usage
> around the same time as the ueventd bug:
> https://bugs.launchpad.net/touch-preview-images/+bug/1211417
> 
> Note that this is not really about power consumption. Colin King has
> done analysis of power consumption on Touch devices and the biggest
> bang for the buck is ensuring that sensors are turned off when they
> are not needed, not minimising CPU usage. Instead, please consider
> this proposal an attempt to better ensure the stability and
> performance of Touch systems out in the wild.
> 
> == Implementation ==
> 
> We will enable the sampling and reporting of high CPU usage in
> background processes on Touch devices when the device is not in
> developer mode.
> 
> Foreground processes will be ignored by this check. They will instead
> be handled by an "application not responding" (ANR) implementation in
> Mir. They will be allowed to use 100% CPU unless they block the UI
> thread for an unreasonable amount of time.
> 
> With the application lifecycle work, background applications will be
> suspended and get no CPU time at all, so this check will only apply to
> system processes.
> 
> Each background process will be periodically sampled for its CPU
> usage. If the process is using a large amount of CPU consistently
> across several of these samplings, it will be killed and an apport
> report will be created.
> 
> An outstanding question is what the threshold should be for high CPU usage.
> 
> == Where will this check live? ==
> 
> It has been suggested that the task of periodically checking for
> runaway processes live inside a long-running and lightweight C
> process. Whoopsie was suggested as a potential candidate.

Something I ran across recently was that I discovered that ofono was not
sending crash reports via whoopsie.  I was told recently that there are
packaging changes required to enable to this to happen.  If someone
could provide a pointer on how to do this, I'd much appreciate.

So, if we do this, do we need to audit all of our system packages and
ensure that they've all been whoopsie-fied (sic)?

> libprocps was raised as potentially helpful, but James pointed out
> that CPU percentage needed to be calculated by the caller, so another
> approach may prove easier.
> 
> == How will we group reports of the same underlying problem? ==
> 
> https://errors.ubuntu.com will need to receive a string that
> represents the problem (a signature) to which this instance of a
> runaway process belongs. This lets the website group together the
> instances of a problem onto a single page and increment the count for
> the problem on the front page leaderboard.
> 
> Whoopsie, or whatever process holds this check, will use the ptrace
> system call to generate a stack trace of the runaway process which
> apport can then use to generate a crash signature:
> http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/apport/report.py#L1199
> 
> Martin suggested we could do three stack traces each 1 s ± <random
> interval> apart, and then chop away the differing part at the top, so
> that we only keep the common bit.
> 
> Since we would be generating multiple stack traces, we cannot just
> build the report and stack trace through the traditional means of
> triggering apport kernel core pipe handler by sending SIGABRT.
> 
> == Bringing the check to the Ubuntu desktop ==
> 
> It was suggested that we could also have this check on the Ubuntu
> desktop, but it was quickly pointed out that great care would need to
> be taken to prevent reporting when gcc or Firefox uses 100% CPU.
> 
> This would be particularly annoying since the desktop currently
> presents a dialog whenever an error occurs. There are plans underway
> to group errors that do not need your immediate attention
> (non-application crashes, e.g. package installation failures) into a
> single dialog with the next error that does require your attention
> (Firefox crashing); however, a quicker solution would be to only
> report these desktop runaway processes on systems that have automatic
> error reporting enabled.
> 
> We could then create a blacklist of processes that are known to be
> intensive but safe using the data gathered from Touch and automatic
> reporting systems and eventually bring reporting of runaway processes
> to all Ubuntu systems (save servers).
> 
> A whitelist was considered, but determined to not save us from
> problems like the ueventd bug.

What about Android's init?  If it's killed, does the lxc Android
container gracefully recover?

Regards,
/tony