yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

I kind of feel this is in the class of bugs of "I hit everything with a
sledge hammer really hard without tuning, and it fell over".

Falling over is what things do with out of the box defaults that are
pushed way beyond their limits.

** Changed in: nova
       Status: New => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1640029

Title:
  [stable/newton] Deleting heat stack failed due to error "QueuePool
  limit of size 50 overflow 50 reached, connection timed out, timeout
  30"

Status in neutron:
  New
Status in OpenStack Compute (nova):
  Won't Fix

Bug description:
  In my stable/newton setup running on a VMware NSX platform, I brought up 5 heat stacks each having 100 nova instances in the same /16 network.
  Deleting those heat stacks failed due to the below error.

  "
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30
  "

  Because of this error, out of 500 instances, deletion of about 67 instances got failed.
  With default parameters in neutron.conf, I'm getting the below neutron error.

  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource [req-a0022887-cc01-4f2e-980d-490136524363 admin -] delete failed: No details.
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource Traceback (most recent call last):
  .
  .
  .
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self._wrap_pool_connect(self.pool.connect, None),
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return fn()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return _ConnectionFairy._checkout(self)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     fairy = _ConnectionRecord.checkout(pool)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     rec = pool._do_get()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     (self.size(), self.overflow(), self._timeout))
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource

  After changing the below parameters in /etc/neutron/neutron.conf and

  max_pool_size = 50
  retry_interval = 10
  max_overflow = 50
  pool_max_size = 50
  pool_max_overflow = 50
  pool_timeout = 30

  below parameters in nova.conf and restarted the services and re-
  executed the testcase.Still deleting heat stack is failing with the
  below error

  max_pool_size = 50
  max_overflow = 50

  n-api.log:

  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in API method
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  .
  .
  .
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     rec = pool._do_get()
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     (self.size(), self.overflow(), self._timeout))
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
  2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.TimeoutError'>
  2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.TimeoutError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1044

  Please look into this.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1640029/+subscriptions