launchpad-dev team mailing list archive

[Seif Lotfy <seif@xxxxxxxxx>]

Thanks for the feedback

On Thu, Nov 18, 2010 at 3:53 AM, Robert Collins <
robert.collins@xxxxxxxxxxxxx> wrote:

> It's late for me right now, but I'm not going to be able to comment
> more until Monday, so I thought I'd give you some brief, quick,
> feedback.
>
> I know that jml is interested in this *sort* of thing eventually in
> Launchpad, so I'm going to limit my response to technical performance
> and scaling considerations.
>
> On Thu, Nov 18, 2010 at 3:37 PM, Seif Lotfy <seif@xxxxxxxxx> wrote:
> > mashed together. So again this is to provide a timeline per
> > project/team/individual.
>
> This suggests doing at least one extra write per write we do, and
> either scatter-gather aggregation on reads or expansion onto related
> objects asynchronously. We have users where this will require
> correlation onto thousands of project+team objects.
>
> > First thing we need for that is a Zeitgeist running on a server with an
> SQL
> > DB different SQLite.
>
> I can't quite parse this, but I think you mean 'different to SQLite'.
> Well, yes, we couldn't run on SQLite. The launchpad production
> database is ~260GB of data. Running on a different database would mean
> either accepting inaccurate data or using some mechanism such as a
> post-commit work queue to load data, or accepting that some updates
> would either (depending on sequencing) not reach zeitgeist or reach
> zeitgeist but not actually happen. Why would we want a different
> database?


No its not different to SQL. I mean PostgreSQL or MySQL. This makes the
changes minimal to Zeitgeist. I think only 2 files need to be changed for
that.


> > This should be no problem.
> > Pushing activity from launchpad into zeitgeist means we need to hook into
> > launchpad and push
> > what the user does into Server-Zeitgeist.
>
> This assumes we know what the user does :) - its reasonably high up my
> 'we should do this' list to have a web API event triggered out of
> Launchpad which things [like Zeitgeist] can hook into.
>

I think its possible to know what the user does or what is happening on
launchpad else I wouldn't be able to see who filed a bug


>
> > Now this is just an example an far from perfect and I would like to
> discuss
> > the idea with you guys here.
> > One of the issues we will face is user privacy. Again that should not be
> an
> > issue since a logic can be
> > implemented between launchpad and Zeitgeist for querying for information.
> > Allowing timelines to be either
> > public or private to the respected teams/individuals.
>
> Privacy (disclosure and access) is a very nuanced topic - its not at
> all amenable to a handwave: I suggest you read through the series of
> topics Curtis has posted about the overall efforts we're entering into
> to make privacy and disclosure in Launchpad rock.
>

Can you please hook me up?


>
> I expect we'd do about 5-10 insertions a second, and if we were to
> make timelines omnipresent then maybe 50 queries a second : how many
> milliseconds does zeitgeist take to answer a query, how much context
> does it need, and how does it scale as you add millions of actions?
> How much disk space would we need to support it per month, and does it
> grow linearly or sublinearly (or worse, does it grow supralinearly?)
>

I will give you a more detailed answer on that tomorrow since we underwent
some major performance improvments. But I think on a good Server we are
pretty quick and scaling should not be an issue at all.

> Please feel free to criticize the idea or help evolve it to something that
> > we can all together work on.
>
> Its not clear to me that we'd want to use zeitgeist as is rather than:
>  - have an zeitgeist API server we could call - loose coupling,
> separated concerns
>  - or implement the algorithms directly against the Launchpad data model
>

Algorithms wont work without a good Log. And Zeitgeist will provide you the
log, the API to access that log and options on working algorithms on top of
this log.


>
> As far as all working together, I'm fine in principle with patches to
> add something like this - as long as jml concurs and we get some
> serious design cycles to user test the impact of adding this to the
> Launchpad web UI, but I very much doubt there are any cycles in the
> work-day for paid Launchpad developers to work on this: we have a
> significant queue of items which are all considered critical by the
> stakeholders who provide planning input to what paid development time
> is spent on.
>
> -Rob
>

I will get into more details tomorrow,
Cheers
Seif