← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1097796] Re: Instances are stuck in deleting task

 

Unfortunately you can't do it via a configuration option, but you should
be able to manually add it to sqlalchemy/session.py

For example:

diff --git a/nova/db/sqlalchemy/session.py b/nova/db/sqlalchemy/session.py
index 3f20632..23c3367 100644
--- a/nova/db/sqlalchemy/session.py
+++ b/nova/db/sqlalchemy/session.py
@@ -99,6 +99,7 @@ def get_engine():

         engine_args = {
             "pool_recycle": FLAGS.sql_idle_timeout,
+            "pool_size": 20,
             "echo": False,
             'convert_unicode': True,
         }

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1097796

Title:
  Instances are stuck in deleting task

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Some instances don't get deleted after I shut them down. They are
  actually still running.

  In nova-compute.log i see the following error

  
  ./nova-compute.log.2:2013-01-07 12:46:48 INFO nova.compute.manager [req-b317e8d1-c0a4-4780-bf38-44a8e61c9e78 3abc796d9c544d039fe7d5b90b206a30 e466feaf9a58472a86989156edc9acf4] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x1deb0d0>| |<nova.rpc.amqp.RpcContext object at 0x46e7cd0>| |c2d827e9-6863-49a7-ad57-576d051f6423|
  ./nova-compute.log.2-2013-01-07 12:46:54 ERROR nova.manager [-] Error during ComputeManager._heal_instance_info_cache: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager Traceback (most recent call last):
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 155, in periodic_tasks
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     task(self, context)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2225, in _heal_instance_info_cache
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     context, self.host)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 578, in instance_get_all_by_host
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     return IMPL.instance_get_all_by_host(context, host)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 102, in wrapper
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     return f(*args, **kwargs)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1515, in instance_get_all_by_host
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     return _instance_get_all_query(context).filter_by(host=host).all()
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     return list(self)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     return self._execute_and_instances(context)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     close_with_result=True)
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
  ./nova-compute.log.2-2013-01-07 12:46:54 TRACE nova.manager     **kw)

  Im using nova-compute 2012.1.1-9. I assume that I need to somehow
  increase QueuePool to fix this issue, how can I do that?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1097796/+subscriptions