yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1212911] Re: database error ?

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1212911@xxxxxxxxxxxxxxxxxx>
Date: Tue, 12 Nov 2013 04:17:35 -0000
Reply-to: Bug 1212911 <1212911@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for neutron because there has been no activity for 60 days.]

** Changed in: neutron
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1212911

Title:
  database error ?

Status in OpenStack Neutron (virtual network service):
  Expired

Bug description:
  Hi list,

  I'm running on CentOS 6.4 with Grizzly,

  after running for several weeks, when I run nova command, such as nova
  boot, nova start and so on, there is always error shows it can't not
  connect to quantum:

  2013-08-16 09:45:56.925 DEBUG nova.api.openstack.wsgi [req-991e44d5-2e27-4d1a-b46e-42f7f3eda4fb e1581aa0be904d0a8384c5f9d9c6891d a169b0ae65eb4d038c43ee747e4690f2] Calling method <bound method Controller.show of <nova.api.openstack.compute.servers.Controller object at 0x2b4eed0>> _process_stack /usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py:911
  2013-08-16 09:46:56.965 ERROR nova.api.openstack [req-991e44d5-2e27-4d1a-b46e-42f7f3eda4fb e1581aa0be904d0a8384c5f9d9c6891d a169b0ae65eb4d038c43ee747e4690f2] Caught error: Request Timeout
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack Traceback (most recent call last):
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 81, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 451, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 890, in __call__
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     content_type, body, accept)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 969, in _process_stack
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     request, action_args)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 863, in post_process_extensions
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     **action_args)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 526, in show
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     return self._show(req, resp_obj)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 522, in _show
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     self._extend_servers(req, [resp_obj.obj['server']])
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 487, in _extend_servers
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     .get_instances_security_groups_bindings(context))
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/network/security_group/quantum_driver.py", line 252, in get_instances_security_groups_bindings
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     ports = quantum.list_ports().get('ports')
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 107, in with_params
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 255, in list_ports
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     **_params)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 997, in list
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 1010, in _pagination
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     res = self.get(path, params=params)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 983, in get
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     headers=headers, params=params)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 968, in retry_request
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     headers=headers, params=params)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 913, in do_request
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack     message=message)
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack QuantumClientException: Request Timeout
  2013-08-16 09:46:56.965 88628 TRACE nova.api.openstack
  2013-08-16 09:46:56.967 INFO nova.api.openstack [req-991e44d5-2e27-4d1a-b46e-42f7f3eda4fb e1581aa0be904d0a8384c5f9d9c6891d a169b0ae65eb4d038c43ee747e4690f2] http://192.168.12.11:8774/v2/a169b0ae65eb4d038c43ee747e4690f2/servers/f8ae5235-e155-4090-8a80-555343bdf57b returned with HTTP 500



  
  So I checked log for quantum server, found error like this :

  
  2013-08-16 09:46:08    ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
  Traceback (most recent call last):
    File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
      rval = self.proxy.dispatch(ctxt, version, method, **args)
    File "/usr/lib/python2.6/site-packages/quantum/common/rpc.py", line 43, in dispatch
      quantum_ctxt, version, method, **kwargs)
    File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
      return getattr(proxyobj, method)(ctxt, **kwargs)
    File "/usr/lib/python2.6/site-packages/quantum/db/dhcp_rpc_base.py", line 185, in update_lease_expiration
      ip_address, lease_remaining)
    File "/usr/lib/python2.6/site-packages/quantum/db/db_base_plugin_v2.py", line 428, in update_fixed_ip_lease_expiration
      fixed_ip.expiration = expiration
    File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 405, in __exit__
      raise
  TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType


  
  Anyone know why this happens ???


  Thanks.
  -chen

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1212911/+subscriptions