launchpad-dev team mailing list archive

[Richard Harding <rick.harding@xxxxxxxxxxxxx>]

On Wed, 04 Jul 2012, Robert Collins wrote:

> On Tue, Jul 3, 2012 at 9:59 AM, Richard Harding
> <rick.harding@xxxxxxxxxxxxx> wrote:
> > On Tue, 03 Jul 2012, Robert Collins wrote:
> >
> >> On Mon, Jul 2, 2012 at 11:56 PM, Richard Harding
> >> <rick.harding@xxxxxxxxxxxxx> wrote:
> >> > The JS combo loader has been enabled behind the feature flag to beta users
> >> > for a bit now and I wanted to ask around if anyone's hit or seen any issues
> >> > that should be addressed before enabling it for all users. If I don't hear
> >> > anything by Wed I'll ask to have that enabled for all users then.
> >>
> >> We've had a couple of unexplained operational incidents where apache
> >> took up a tonne of CPU on the hosts that the combo loader instances
> >> run on. I think it would be prudent to dig into that a bit before
> >> expanding the user base.
> >>
> >> -Rob
> >
> > Thanks for the heads up. I didn't see anything in the Team Lead email and
> > not seeing anything in the incident report list. If you get a chance, can
> > you help point me in the right direction of what I should be looking
> > at/into to check this out?
>
> Was this before or after we chatted on IRC? If after - its in the
> incident log, not a full blown report. (But perhaps we should have
> done one:)).
>
> -Rob

This was before we had a chat on irc. I had a chat with on irc and put
together a list of the information we have so far.

https://docs.google.com/a/canonical.com/document/d/1id91-eIKP_f6gJFAWmbyNtxwKxcNygOsJH2qOZqUP2c/edit

I've asked webops to debug/trace into the issue if it happens again as I've
been unable to find anything to act upon yet.

However, I do want to note that while working on a different project we had
an apache issue that was thought to possibly have been from the combo
loader. At that time, we were using a tool to run the wsgi app through the
Django application and we weren't sure what was up. It was on a staging
server and only happened once so we didn't get much info out of the
situation.

Give this I'd like to get your ok to change the feature flag to enable the
combo loader for all users. Overall, the combo loader has been enabled for
Launchpad devs since 5/14 and for beta users since 6/12 and it appears
there's the one issue that might be related to keep an eye on.

--

Rick Harding

Launchpad Developer
https://launchpad.net/~rharding
@mitechie