launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

It's late for me right now, but I'm not going to be able to comment
more until Monday, so I thought I'd give you some brief, quick,
feedback.

I know that jml is interested in this *sort* of thing eventually in
Launchpad, so I'm going to limit my response to technical performance
and scaling considerations.

On Thu, Nov 18, 2010 at 3:37 PM, Seif Lotfy <seif@xxxxxxxxx> wrote:
> mashed together. So again this is to provide a timeline per
> project/team/individual.

This suggests doing at least one extra write per write we do, and
either scatter-gather aggregation on reads or expansion onto related
objects asynchronously. We have users where this will require
correlation onto thousands of project+team objects.

> First thing we need for that is a Zeitgeist running on a server with an SQL
> DB different SQLite.

I can't quite parse this, but I think you mean 'different to SQLite'.
Well, yes, we couldn't run on SQLite. The launchpad production
database is ~260GB of data. Running on a different database would mean
either accepting inaccurate data or using some mechanism such as a
post-commit work queue to load data, or accepting that some updates
would either (depending on sequencing) not reach zeitgeist or reach
zeitgeist but not actually happen. Why would we want a different
database?

> This should be no problem.
> Pushing activity from launchpad into zeitgeist means we need to hook into
> launchpad and push
> what the user does into Server-Zeitgeist.

This assumes we know what the user does :) - its reasonably high up my
'we should do this' list to have a web API event triggered out of
Launchpad which things [like Zeitgeist] can hook into.

> Now this is just an example an far from perfect and I would like to discuss
> the idea with you guys here.
> One of the issues we will face is user privacy. Again that should not be an
> issue since a logic can be
> implemented between launchpad and Zeitgeist for querying for information.
> Allowing timelines to be either
> public or private to the respected teams/individuals.

Privacy (disclosure and access) is a very nuanced topic - its not at
all amenable to a handwave: I suggest you read through the series of
topics Curtis has posted about the overall efforts we're entering into
to make privacy and disclosure in Launchpad rock.

I expect we'd do about 5-10 insertions a second, and if we were to
make timelines omnipresent then maybe 50 queries a second : how many
milliseconds does zeitgeist take to answer a query, how much context
does it need, and how does it scale as you add millions of actions?
How much disk space would we need to support it per month, and does it
grow linearly or sublinearly (or worse, does it grow supralinearly?)

> Please feel free to criticize the idea or help evolve it to something that
> we can all together work on.

Its not clear to me that we'd want to use zeitgeist as is rather than:
 - have an zeitgeist API server we could call - loose coupling,
separated concerns
 - or implement the algorithms directly against the Launchpad data model

As far as all working together, I'm fine in principle with patches to
add something like this - as long as jml concurs and we get some
serious design cycles to user test the impact of adding this to the
Launchpad web UI, but I very much doubt there are any cycles in the
work-day for paid Launchpad developers to work on this: we have a
significant queue of items which are all considered critical by the
stakeholders who provide planning input to what paid development time
is spent on.

-Rob