launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Tue, Nov 29, 2011 at 11:20 AM, Martin Pool <mbp@xxxxxxxxxxxxxx> wrote:
> On 28 November 2011 19:46, Robert Collins <robertc@xxxxxxxxxxxxxxxxx> wrote:
>> print "create temporary table temp_topic_events (topic integer, event
>> integer, date timestamp );"
>> print "begin;"
>> for event in xrange(10000000):
>>    topic = int(random.random() * 10000)
>>    days_old = int(random.random() * 720)
>>    print "insert into temp_topic_events values (%s, %s, now() - '%s
>> days'::interval);" % (topic, event, days_old)
>
> I realize I can do that.  My concern was more that (in fact as you
> told me just the other day) local load tests are poorly correlated
> with how something will work with realistically-distributed data under
> production load on production hardware. Also, that RTs are very

Indeed. In this case, I've run that script against staging, to get
/some/ sense of performance.

Its a very narrow table (width 16) ; it can pull back the first 75
events for the last 6 months of a random topic in 3ms (with an
additional index). It can bring back the last 75 global events cold in
60ms [measured by leaving it overnight to get swapped out :P].

> backlogged (I just touched one that's 10 months old), and all else
> being equal something that can make progress without requiring
> sysadmin changes is desirable.

I agree subject to the all else being equal caveat :)

> But it's stupid to argue that something is difficult.  Eventually it
> probably does want a separate db; and that db may be easier to scale
> and manage than one huge one.

Yes. Also LEAN says one should do the things that are difficult more
often to make them easy.

-Rob