yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1268423] [NEW] multiple api_wokers cannot connect to pgsql

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: gustavo panizzo <1268423@xxxxxxxxxxxxxxxxxx>
Date: Mon, 13 Jan 2014 00:27:59 -0000
Reply-to: Bug 1268423 <1268423@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when api_workers > 0 i got a TRACE in the logs, neutron server works,
but only one server appears to be connected to db and working

2014-01-12 21:21:52.639 10512 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     **args)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 44, in dispatch
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 186, in report_state
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     self.plugin.create_or_update_agent(context, agent_state)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 149, in create_or_update_agent
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context, agent['agent_type'], agent['host'])
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 125, in _get_agent_by_type_and_host
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     Agent.host == host).one()
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     ret = list(self)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     return self._execute_and_instances(context)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     result = conn.execute(querycontext.statement, self._params)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     params)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     compiled_sql, distilled_params
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     cursor.execute(statement, parameters)
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp OperationalError: (OperationalError) SSL connection has been closed unexpectedly
2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp  'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents."binary" AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations \nFROM agents \nWHERE agents.agent_type = %(agent_type_1)s AND agents.host = %(host_1)s' {'host_1': u'kvm03d', 'agent_type_1': u'Open vSwitch agent'}


i can reproduce the issue running on devstack with postgresl, if i
switch to mysql it works fine

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: havana

** Tags added: havana

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1268423

Title:
  multiple api_wokers cannot connect to pgsql

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  when api_workers > 0 i got a TRACE in the logs, neutron server works,
  but only one server appears to be connected to db and working

  2014-01-12 21:21:52.639 10512 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     **args)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 44, in dispatch
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 186, in report_state
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     self.plugin.create_or_update_agent(context, agent_state)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 149, in create_or_update_agent
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context, agent['agent_type'], agent['host'])
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 125, in _get_agent_by_type_and_host
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     Agent.host == host).one()
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     ret = list(self)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     return self._execute_and_instances(context)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     result = conn.execute(querycontext.statement, self._params)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     params)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     compiled_sql, distilled_params
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     context)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp     cursor.execute(statement, parameters)
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp OperationalError: (OperationalError) SSL connection has been closed unexpectedly
  2014-01-12 21:21:52.639 10512 TRACE neutron.openstack.common.rpc.amqp  'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents."binary" AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations \nFROM agents \nWHERE agents.agent_type = %(agent_type_1)s AND agents.host = %(host_1)s' {'host_1': u'kvm03d', 'agent_type_1': u'Open vSwitch agent'}


  i can reproduce the issue running on devstack with postgresl, if i
  switch to mysql it works fine

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1268423/+subscriptions
Follow ups

[Bug 1268423] Re: multiple api_wokers cannot connect to pgsql
From: Thierry Carrez, 2014-06-12
[Bug 1268423] [NEW] multiple api_wokers cannot connect to pgsql
From: gustavo panizzo, 2014-01-13
References

[Bug 1268423] [NEW] multiple api_wokers cannot connect to pgsql
From: gustavo panizzo, 2014-01-13