openstack team mailing list archive

[Monty Taylor <mordred@xxxxxxxxxxxx>]

Whoops!

Ok- turns out I was running against system installed libs instead of
trunk. doh.

I can now not reproduce your issues with trunk. I spawned off 10
concurrent copies of that program. Now, some of them weren't able to get
ips any more (since it's only configured for a pool of five of them.) -
but I can no longer trigger the AttributeError: 'NoneType' object has no
attribute '_keymap' traceback.



On 09/25/2011 05:48 PM, Monty Taylor wrote:
> What was the intent of the connection pooling? That is, what was it
> trying to fix?
> 
> Running the script a single time caused 16 connection threads to be
> spawned. This seems a bit excessive.
> 
> When I tried spawning five copies at the same time, I wound up with 60
> connected threads, plus some connect timeouts, plus some of the
> tracebacks you're seeing below.
> 
> Increasing the thread_cache_size from the default on ubuntu (which is 8)
> helped things, but things still seemed to be going to the bad place.
> 
> More vexing is that all of these queries were doing select ... for
> update (makes sense why) - but that just means that we're stacking up on
> locks in the db trying to get these resources... one of those situations
> where greater parallelism actually isn't better.
> 
> Without really knowing more, my vote would be not to have app-level
> connection pooling by default. MySQL specifically doesn't really like
> having a bunch of unused connection sitting around for long lifecycles
> (with a few exceptions - the exceptions always prove the rule, of course)
> 
> Of course- I could be wrong... which is why I'd like to know more about
> what the issue was that incited connection pooling.
> 
> Monty
> 
> On 09/25/2011 01:53 PM, Vishvananda Ishaya wrote:
>> Hey everyone,
>>
>> I'm a bit concerned with the connection pooling in the db.  It seems that things are not getting cleaned up properly.  I have a repro-case that causes failures that we have seen before.  if I revert the nova/db/sqlalchemy/session.py to before the eventlet db pool was added I get no failures.  If you want to see the issue, try the attached code.  You will need to run from the nova directory or do python setup.py develop.  You will also need to create a mysql database called test and edit the sql_connection string if you have a mysql password, etc.  Please check this code.  If we can't come up with a fix, I htink we need to revert back to no connection pooling.
>>
>> Run the attached script at least 3 times The code below runs fine the first couple of times, Then it starts to fail with the following error:
>>
>> 2011-09-24 12:51:02,799 INFO sqlalchemy.engine.base.Engine.0x...36d0 [-] ROLLBACK
>> Traceback (most recent call last):
>>  File "/Library/Python/2.7/site-packages/eventlet/hubs/hub.py", line 336, in fire_timers
>>    timer()
>>  File "/Library/Python/2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
>>    cb(*args, **kw)
>>  File "/Library/Python/2.7/site-packages/eventlet/event.py", line 163, in _do_send
>>    waiter.switch(result)
>>  File "/Library/Python/2.7/site-packages/eventlet/greenthread.py", line 192, in main
>>    result = function(*args, **kwargs)
>>  File "dbrepro.py", line 44, in associate
>>    ip = db.fixed_ip_associate_pool(ctxt, 1, instance_id=val)
>>  File "/Users/vishvananda/os/nova/nova/db/api.py", line 352, in fixed_ip_associate_pool
>>    instance_id, host)
>>  File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 102, in wrapper
>>    return f(*args, **kwargs)
>>  File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 725, in fixed_ip_associate_pool
>>    filter_by(host=None).\
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1496, in first
>>    ret = list(self[0:1])
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1405, in __getitem__
>>    return list(res)
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1669, in instances
>>    fetch = cursor.fetchall()
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line 2383, in fetchall
>>    l = self.process_rows(self._fetchall_impl())
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line 2366, in process_rows
>>    keymap = metadata._keymap
>> AttributeError: 'NoneType' object has no attribute '_keymap'
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>