openstack team mailing list archive

[Vishvananda Ishaya <vishvananda@xxxxxxxxx>]

We were using the eventlet db pool class.

You can see the code here:

https://github.com/openstack/nova/blob/2011.3/nova/db/sqlalchemy/session.py

there were a few bugs about it, but here is one:

https://bugs.launchpad.net/nova/+bug/838581

Vish

On Mar 1, 2012, at 6:11 PM, Kapil Thangavelu wrote:

> 
> I actually didn't mean to send the parent.. but since we're here. 
> 
> Additional threads/pool make the problem potentially worse, they are the origin 
> of the greenlet switching issue that was the start of this thread. ie. with the 
> monkey patching, most stdlib socket, thread module usage by python code in the 
> non-main threads will potentially attempt a greenlet trampoline across threads 
> at worst (ie. error), or unintentional per thread hubs at best. 
> 
> Since the mysqldb is an extension (no python code) wrapping it in eventlet 
> thread pool should work in theory. What where the problems with it last time it 
> was attempted?
> 
> cheers,
> 
> -kapil
> 
> 
> Excerpts from Devin Carlen's message of 2012-03-01 20:38:20 -0500:
>> As long as we allocate a thread in the eventlet thread pool for the number of mysql connections we want to actually maintain in our connection pool, we shouldn't have problems getting the results we want even with the blocking mysql c drivers. 
>> 
>> Devin
>> 
>> On Thursday, March 1, 2012 at 5:23 PM, Kapil Thangavelu wrote:
>> 
>>> The standard python postgresql driver (psycopg2) does have an async mode. There 
>>> are non db api compliant async mysql drivers for gevent.
>>> 
>>> 
>>> Excerpts from Vishvananda Ishaya's message of 2012-03-01 15:36:43 -0500:
>>>> Yes it does. We actually tried to use a pool at diablo release and it was very broken. There was discussion about moving over to a pure-python mysql library, but it hasn't been tried yet.
>>>> 
>>>> Vish
>>>> 
>>>> On Mar 1, 2012, at 11:45 AM, Yun Mao wrote:
>>>> 
>>>>> There are plenty eventlet discussion recently but I'll stick my
>>>>> question to this thread, although it's pretty much a separate
>>>>> question. :)
>>>>> 
>>>>> How is MySQL access handled in eventlet? Presumably it's external C
>>>>> library so it's not going to be monkey patched. Does that make every
>>>>> db access call a blocking call? Thanks,
>>>>> 
>>>>> Yun
>>>>> 
>>>>> On Wed, Feb 29, 2012 at 9:18 PM, Johannes Erdfelt <johannes@xxxxxxxxxxx (mailto:johannes@xxxxxxxxxxx)> wrote:
>>>>>> On Wed, Feb 29, 2012, Yun Mao <yunmao@xxxxxxxxx (mailto:yunmao@xxxxxxxxx)> wrote:
>>>>>>> Thanks for the explanation. Let me see if I understand this.
>>>>>>> 
>>>>>>> 1. Eventlet will never have this problem if there is only 1 OS thread
>>>>>>> -- let's call it main thread.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> In fact, that's exactly what Python calls it :)
>>>>>> 
>>>>>>> 2. In Nova, there is only 1 OS thread unless you use xenapi and/or the
>>>>>>> virt/firewall driver.
>>>>>>> 3. The python logging module uses locks. Because of the monkey patch,
>>>>>>> those locks are actually eventlet or "green" locks and may trigger a
>>>>>>> green thread context switch.
>>>>>>> 
>>>>>>> Based on 1-3, does it make sense to say that in the other OS threads
>>>>>>> (i.e. not main thread), if logging (plus other pure python library
>>>>>>> code involving locking) is never used, and we do not run a eventlet
>>>>>>> hub at all, we should never see this problem?
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> That should be correct. I'd have to double check all of the monkey
>>>>>> patching that eventlet does to make sure there aren't other cases where
>>>>>> you may inadvertently use eventlet primitives across real threads.
>>>>>> 
>>>>>> JE
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx (mailto:openstack@xxxxxxxxxxxxxxxxxxx)
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx (mailto:openstack@xxxxxxxxxxxxxxxxxxx)
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx (mailto:openstack@xxxxxxxxxxxxxxxxxxx)
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help : https://help.launchpad.net/ListHelp
>>> 
>>>