launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

Hi, all.

On Wed, Aug 4, 2010 at 7:10 AM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> On Wed, Aug 4, 2010 at 2:46 PM, Robert Collins
> <robert.collins@xxxxxxxxxxxxx> wrote:
>> So, we've got a partial deployment of memcache in place, and while it
>> may be contentious, I think it has distracted us from solving root
>> cause problems.
> ..> So, I propose to turn memcached off on Friday, unless I'm lynched
>> before then, which I kindof expect :)
>
> For clarity, I'm proposing removing all the uses of it that users are
> noticing (and thus complaining about) - not removing the facility from
> the datacentre: memcached is a good tool and its good that we have it
> available.
>

Rob and I chatted on IRC about this thread and my concerns.  I'll just
record them here for the sake of people watching this thread but not
IRC.

In the end, I'm happy with Rob's final proposal above -- that we
should remove unnecessary or bad uses of memcached and that were not
focusing on wide spread use until we fix root causes of timeout
issues.  The facility to use memcached would remain in place.  I
originally read this as "rip out memcached completely" which had me
concerned.  I understand Rob is not pushing for that now.  I do,
however, hate to see us move away from memcached completely.  It's a
great tool when used properly, and like feature flags, we need to be
using it to gain experience, facility, and expertise.

I took some issue with the hot bugs list being held up as the poster
child for bad memcached usage and the assumption that the use of
caching there was to fix timeout issues.  Go look at bug 590992.
Adding in caching was done only after the initial timeout issue was
fixed, and this was done during the great DB fire post-10.06 rollout,
and only as a means to buy some overhead when high concurrency or high
load raised it's ugly head again.  I was never happy that the data was
stale as bugs in the list were worked on, but my assumption, based on
bug comments I think, was that we were still refining the memcached
mechanism and that some form of cache invalidation was being worked
on, which is why I didn't back out the caching there yet.  Certainly,
I'm fine removing the caching of this list if there's nothing being
worked on to make invalidating cached fragments possible.

IMHO, we can refine our use of memcached while also developing a
shared understanding of how best to fix timeouts and performance
issues without discounting or removing all the work that has gone on
before.  Also, I really appreciate Foundations providing memcached and
know it will be a great benefit when we get its use correct.


Cheers,
deryck



-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/