openstack team mailing list archive

[Monty Taylor <mordred@xxxxxxxxxxxx>]

Because I'm just going to spam the list all afternoon...

Out of curiousity, I ramped up the numbers in the script to get a most
sustained attack on the db. With the old db code (pre pool) - running 10
concurrent copies of dbrepro.py gave me 10 db connections (as you'd
expect) and a consistent sustained throughput of 45 qps for a while.
When I pulled forward to trunk, well - the first time I hit out of
connections, because I was configured with max_connections of 150. Once
I upped that and restarted mysql, I got 185 concurrent connections to
the database (why 185? I don't know) with bursts of 10 queries a second
once every 5 seconds with a single burst on the front end of 101 queries
in a second. (oh, and btw - with trunk and 10 concurrent copies I
started getting that traceback again)

fwiw

On 09/25/2011 05:59 PM, Monty Taylor wrote:
> Hrm. It's not piling on with the locking like I originally thought -
> from the db's perspective there's not a whole hell of a lot going on,
> actually - even two copies of this script run concurrently cause them to
> totally get in to some app-level spinning that ends up with at least one
> script erroring. I'll keep poking, mainly just because it's interesting
> - but it's certainly not happy.
> 
> On 09/25/2011 01:53 PM, Vishvananda Ishaya wrote:
>> Hey everyone,
>>
>> I'm a bit concerned with the connection pooling in the db.  It seems that things are not getting cleaned up properly.  I have a repro-case that causes failures that we have seen before.  if I revert the nova/db/sqlalchemy/session.py to before the eventlet db pool was added I get no failures.  If you want to see the issue, try the attached code.  You will need to run from the nova directory or do python setup.py develop.  You will also need to create a mysql database called test and edit the sql_connection string if you have a mysql password, etc.  Please check this code.  If we can't come up with a fix, I htink we need to revert back to no connection pooling.
>>
>> Run the attached script at least 3 times The code below runs fine the first couple of times, Then it starts to fail with the following error:
>>
>> 2011-09-24 12:51:02,799 INFO sqlalchemy.engine.base.Engine.0x...36d0 [-] ROLLBACK
>> Traceback (most recent call last):
>>  File "/Library/Python/2.7/site-packages/eventlet/hubs/hub.py", line 336, in fire_timers
>>    timer()
>>  File "/Library/Python/2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
>>    cb(*args, **kw)
>>  File "/Library/Python/2.7/site-packages/eventlet/event.py", line 163, in _do_send
>>    waiter.switch(result)
>>  File "/Library/Python/2.7/site-packages/eventlet/greenthread.py", line 192, in main
>>    result = function(*args, **kwargs)
>>  File "dbrepro.py", line 44, in associate
>>    ip = db.fixed_ip_associate_pool(ctxt, 1, instance_id=val)
>>  File "/Users/vishvananda/os/nova/nova/db/api.py", line 352, in fixed_ip_associate_pool
>>    instance_id, host)
>>  File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 102, in wrapper
>>    return f(*args, **kwargs)
>>  File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 725, in fixed_ip_associate_pool
>>    filter_by(host=None).\
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1496, in first
>>    ret = list(self[0:1])
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1405, in __getitem__
>>    return list(res)
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line 1669, in instances
>>    fetch = cursor.fetchall()
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line 2383, in fetchall
>>    l = self.process_rows(self._fetchall_impl())
>>  File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line 2366, in process_rows
>>    keymap = metadata._keymap
>> AttributeError: 'NoneType' object has no attribute '_keymap'
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>