launchpad-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxxxxxx>]

On Tue, Aug 3, 2010 at 11:02 AM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> On Tue, Aug 3, 2010 at 12:49 PM, Robert Collins
> <robert.collins@xxxxxxxxxxxxx> wrote:
>> Just a reminder - I'm spending today, all day, just poking at
>> performance; I'm 100% interruptible for performance related questions
>> / tasks / todos & gotchas.
>>
>> And if anyone wants to join in, please do so!
>>
>> Right now, I'm about to finish up a small backlog of emails, then look
>> into why https://api.staging.launchpad.net/beta/~ubuntu-dev/participants
>> takes 8 seconds.
>
> So, I'm heading off to sleep, and I haven't got a mergable branch yet,
> though I do have some small test cleanups.
>
> Largely I've been on a voyage of discovery of a few bits of the
> system, which is ok :).
>

Thanks for sending these notices, btw. It's nice to see what's going on.

...
> I grabbed a profile off of staging. This required some bootstrapping
> of that facility. Remember the excellent kcachegrind stuff Maris wrote
> about? Well we can get that from staging, with LOSA cooperation. Its a
> little manual at the moment, so its worth seeing if you can reproduce
> things locally first.
>
> To get a profile from staging, follow these basic steps:
>  - grab a LOSA
>  - ask them to set profile_requests: True in the staging config (on
> staging) and bounce the appserver to make it take effect. I have a
> branch that makes this trivial which I will get merged right after the
> rollout; for now you need to apply the following:
> === modified file 'staging-lazr.conf'
> --- staging-lazr.conf   2010-07-19 06:53:44 +0000
> +++ staging-lazr.conf   2010-08-03 01:02:11 +0000
> @@ -246,3 +246,9 @@
>  [calculate_bug_heat]
>  oops_prefix: CBHT
>  error_dir: /srv/staging.launchpad.net/staging-logs/calculate_bug_heat
> +
> +[profiling]
> +profile_requests: True
> +profile_dir: /srv/staging.launchpad.net/staging-logs/profiling
>
> With that applied, every request will generate a prof file
> (datestamp-pageid-serial.prof) in
> /srv/staging.launchpad.net/staging-logs/profiling. So, hit your page
> up, and then
>  - ask your friendly LOSA to revert the config and bounce the appserver again
>  - ask them to rsync the logs/oops etc to devpad
>
> You can then rsync the profile down locally - rsync -r
> devpad.canonical.com:/srv/launchpad.net-logs/staging/asuka/profiling/
> SOMEPATH -. To find the profile you want, look for the datestamp
> matching your web request and the page id. This can be a bit fiddly
> :(. There should be a oops comment in the page if you hit ctrl-U, but
> API pages (which I was looking at) don't render that.
>
> Anyhow, having found it (thanks Steve!) note that the name - e.g.
> 2010-08-03_02:04:43.893-MailingListApplication:MailingListAPIView-OOPS-1676S219-Dummy-4.prof
> - has an OOPS code in it. You can look this up on
> lp-oops.canonical.com, and correlate the SQL activity and the python
> backtrace together.
>

This sounds like a guide that should be in a doc somewhere, perhaps a
wikipage. Could you please do so?

I would probably have looked for it in the debugging section of the dev wiki.

...
> Now, given Tim has run into this in code, it may be that there is a
> general problem pattern we should look into, but we should also fix
> this up. In the code API, what they do is a tuple query in storm to
> get back all the needed information and then start yielding
> pre-populated objects which already have the needed values looked up.
> This is a bit manual and error prone but a sensible approach for now.
>

Where can we find this code?

> I hope this slightly rambling story is useful - its probably the case
> that its all all documented already and I just missed it.
>

I think this points to a more general problem with documentation
navigation. I think the apidoc work done recently by Salgado & Gary is
an excellent first step along these lines.

> If not though, please do tell me where you would naively have looked
> for the interesting bits of the story, and I'll put them there
> (wherever there is) in a more permanent fashion.
>

As above.

jml