yahoo-eng-team team mailing list archive

[Qin Zhao <chaochin@xxxxxxxxx>]

** Also affects: keystone
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1389985

Title:
  CLI will fail one time after restarting DB

Status in OpenStack Telemetry (Ceilometer):
  New
Status in OpenStack Image Registry and Delivery Service (Glance):
  In Progress
Status in Orchestration API (Heat):
  New
Status in OpenStack Identity (Keystone):
  New
Status in OpenStack Compute (Nova):
  New
Status in Oslo Database library:
  New

Bug description:
  After restarting database, the first command will fail. for example:
  after restarting Database, and wait for a few minutes.
  Then run heat stack-list, result will be like below:

  ERROR: Remote error: DBConnectionError (OperationalError) ibm_db_dbi::OperationalError: SQLNumResultCols failed: [IBM][CLI Driver] SQL30081N  A communication error has been detected. Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  Location where the error was detected: "10.11.1.14".  Communication function detecting the error: "send".  Protocol specific error code(s): "2", "*", "*".  SQLSTATE=08001 SQLCODE=-30081 'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = ? ORDER BY stack.created_at DESC, stack.id DESC' ('a3a14c6f82bd4ce88273822407a0829b',)
  [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/usr/lib/python2.6/site-packages/heat/engine/service.py", line 69, in wrapped\n    return func(self, ctx, *args, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/heat/engine/service.py", line 490, in list_stacks\n    return [api.format_stack(stack) for stack in stacks]\n', u'  File "/usr/lib/python2.6/site-packages/heat/engine/stack.py", line 264, in load_all\n    show_deleted, show_nested) or []\n', u'  File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 130, in stack_get_all\n    show_deleted, show_nested)\n', u'  File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 368, in stack_get_all\n    marker, sort_dir, filters).all()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all\n    return list(self)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances\n    result = conn.execute(querycontext.statement, self._params)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute\n    params)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context\n    context)\n', u'  File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception\n    six.reraise(type(newraise), newraise, sys.exc_info()[2])\n', u'  File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception\n    per_fn = fn(ctx)\n', u'  File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler\n    context.is_disconnect)\n', u'  File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 263, in _is_db_connection_error\n    raise exception.DBConnectionError(operational_error)\n', u'DBConnectionError: (OperationalError) ibm_db_dbi::OperationalError: SQLNumResultCols failed: [IBM][CLI Driver] SQL30081N  A communication error has been detected. Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  Location where the error was detected: "10.11.1.14".  Communication function detecting the error: "send".  Protocol specific error code(s): "2", "*", "*".  SQLSTATE=08001 SQLCODE=-30081 \'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.updated_at AS stack_updated_at \\nFROM stack \\nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = ? ORDER BY stack.created_at DESC, stack.id DESC\' (\'a3a14c6f82bd4ce88273822407a0829b\',)\n'].

  then run heat stack-list or other command, the command will be ok.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1389985/+subscriptions