launchpad-dev team mailing list archive

[Michael Hudson-Doyle <michael.hudson@xxxxxxxxxxxxx>]

On Thu, 01 Sep 2011 17:46:53 -0400, "Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx> wrote:
Non-text part: multipart/signed
> Sorry to crash the party, but the librarian isn't using a connection
> pool object at all. I remember it seeing mention of tha,  but I think
> this was removed very long ago. At least ever since it uses the standard
> zopeless DB code. We are simply calling deferToThread for code that
> calls the DB.

Ah, indeed.

> We don't explicitely configure a thread pool either. So it's either
> unbounded or using twisted default settings. And looking at
> deferToThread it seems that as many threads as required will be created:
> 
> def deferToThread(f):
>     """Run the given callable in a separate thread and return a Deferred
> which
>     fires when the function completes.
>     """
>     def decorated(*args, **kwargs):
>         d = defer.Deferred()
>         def runInThread():
>             return threads._putResultInDeferred(d, f, args, kwargs)
> 
>         t = threading.Thread(target=runInThread)
>         t.start()
>         return d
>     return mergeFunctionMetadata(f, decorated)

Where did you find that deferToThread[1]?  AFAICT, this is the one used
by the librarian (from
eggs/Twisted-11.0.0-py2.6-linux-x86_64.egg/twisted/internet/threads.py):

def deferToThread(f, *args, **kwargs):
    """
    Run a function in a thread and return the result as a Deferred.

    @param f: The function to call.
    @param *args: positional arguments to pass to f.
    @param **kwargs: keyword arguments to pass to f.

    @return: A Deferred which fires a callback with the result of f,
    or an errback with a L{twisted.python.failure.Failure} if f throws
    an exception.
    """
    from twisted.internet import reactor
    return deferToThreadPool(reactor, reactor.getThreadPool(),
                             f, *args, **kwargs)

So, we're using the default thread pool, and that seems to be configured
to use a maximum of 10 threads:

        def _initThreadPool(self):
            """
            Create the threadpool accessible with callFromThread.
            """
            from twisted.python import threadpool
            self.threadpool = threadpool.ThreadPool(
                0, 10, 'twisted.internet.reactor')

Cheers,
mwh

[1] we should kill it!