launchpad-dev team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

On Tue, May 18, 2010 at 3:36 AM, Curtis Hovey
<curtis.hovey@xxxxxxxxxxxxx> wrote:
> I have been looking at timeout issues where the page has lots of content
> and lots of db lookups, such as large milestone listings. I think I have
> a  plan so clever it you could put a tale on it can call it a fox.
>
> I compared the db calls between a two milestones with 360 bugs, one with
> assignees, the other with none -- 50 verses 440 calls. Why, icons baby!
>
> We do icon lookups when we render pillars and person links. How often
> does a pillar or team change its icon? I want to cache this for a day,
> but I will settle for an hour if people are uneasy. (I do remember that
> 36 teams are private and I can provide caching rules for them)
>
> The tricky part is that Stuart's memcached integration is in page the
> template. We need to provide rules that our tales helpers can use. Is it
> worth while? Are memcached look ups cheaper than db lookups?

Memcached lookups are much faster than DB lookups. I think 2-3 times
faster for the most trivial.

I think the page template syntax is complete. If caching doesn't fit
in there, you should probably use the IMemcacheClient utility
directly:

client = getUtility(IMemcacheClient)
key = "lp:%s:%s" % (domain, key)
value = client.get(key)
if value is None:
    value = calculate_expensive_value()
    client.set(key, value)
return value


At the moment, the IMemcacheClient doesn't define an API - you can use
the methods provided by the underlying Python memcache client
(eggs/python_memcached-1.45-py2.5.egg/memcache.py). If we have more
things using the lower level API, we should add the methods we want to
support to the IMemcacheClient interface.

At this stage, I don't think we should be using more than the basic
get and set methods as this means we can trivially replace memcached
with an alternative cache such as Cassandra, MongoDB etc. Also note
that None is returned for a failed lookup which makes it difficult to
store None values in the cache, which sucks but I'm not sure if it is
worth wrapping the Python memcache library to transparently handle
None and have the get method raise a LookupError on a cache miss.


-- 
Stuart Bishop <stuart@xxxxxxxxxxxxxxxx>
http://www.stuartbishop.net/