← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #13855

[Bug 1312002] [NEW] nova cell-show <cell name> fails in devstack development environment

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I am using devstack development environment with n-cell enabled in localrc.       
When I call API  to list all cells a cell with name "child" is explored.
Again when I try to "cell-show" this cell "child" call is failed with an error response 500.
following is the operations followed - 

 Setting in /etc/nova/nova.conf
         
        [cells]
        name = region
        cell_type = api
        enable = True
         
         


When API call to get cells -

REQUEST -          
        curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token:<token>"

RESPONSE - 
        HTTP/1.1 200 OK
        Content-Type: application/json
        Content-Length: 111
        X-Compute-Request-Id: req-e83c337a-a7d9-47e5-a472-72cd41126f21
        Date: Thu, 24 Apr 2014 04:12:34 GMT
         
        {"cells": [{"username": "guest", "rpc_host": "10.0.9.40", "type": "child", "name": "child", "rpc_port": 5672}]}
         
         
         
During particular cell show (using cell name as "child")-
         
REQUEST -
        curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells/child' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: <token>"
         
RESPONSE - 
        HTTP/1.1 500 Internal Server Error
        Content-Length: 128
        Content-Type: application/json; charset=UTF-8
        X-Compute-Request-Id: req-ba2503bd-3948-4cce-8ee6-543aa597497b
        Date: Thu, 24 Apr 2014 04:10:33 GMT
         
        {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
         


Note: when using cell name as "api" or "region" 404 is returned.
     
     
     
     
nova-api logs during show call is traced as below -
     
    2014-04-24 11:15:20.219 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token from (pid=705) __call__ /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:603                                                                                                                    
    2014-04-24 11:15:20.220 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role from (pid=705) _remove_auth_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:662                                                                                                                                                          
    2014-04-24 11:15:20.232 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache from (pid=705) _cache_put /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:1121                                                                                                                    
    2014-04-24 11:15:20.233 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: 6cf0c59310fb4b189fc157b19d0e1026 with project_id : 2c3857a83f454b7cb073891ef47acd11 and roles: heat_stack_owner,admin  from (pid=705) _build_user_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:910                                                                                                                                              
    2014-04-24 11:15:20.238 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Matched GET /2c3857a83f454b7cb073891ef47acd11/os-cells/child from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
    2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Route path: '/{project_id}/os-cells/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102                                                                                                                                                
    2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>, 'project_id': u'2c3857a83f454b7cb073891ef47acd11', 'id': u'child'} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103                                                                                                                      
    2014-04-24 11:15:20.240 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Calling method '<bound method Controller.show of <nova.api.openstack.compute.contrib.cells.Controller object at 0x3789c10>>' (Content-type='None', Accept='application/json') from (pid=705) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:945                                                                                                                
    2014-04-24 11:15:20.253 ERROR nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Caught error: Circular reference detected
    Traceback (most recent call last):
     
      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
        incoming.message))
     
      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
        self._send_reply(conn, reply, failure, log_failure=log_failure)
     
      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
        conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
     
      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
        _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
     
      File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
        return json.dumps(value, default=default, **kwargs)
     
      File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
        **kw).encode(obj)
     
      File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
        chunks = self.iterencode(o, _one_shot=True)
     
      File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
        return _iterencode(o, 0)
     
    ValueError: Circular reference detected
     
    2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return req.get_response(self.application)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     application, catch_exc_info=False)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 616, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.app(env, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     response = self.app(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     content_type, body, accept)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return method(req=request, **action_args)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/common.py", line 602, in inner
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return function(*args, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/cells.py", line 264, in show
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     cell = self.cells_rpcapi.cell_get(context, id)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/cells/rpcapi.py", line 450, in cell_get
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     wait_for_reply=True, timeout=timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     timeout=timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self._send(target, ctxt, message, wait_for_reply, timeout)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     raise result
    2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
    2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     incoming.message))
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     self._send_reply(conn, reply, failure, log_failure=log_failure)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return json.dumps(value, default=default, **kwargs)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     **kw).encode(obj)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     chunks = self.iterencode(o, _one_shot=True)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    2014-04-24 11:15:20.253 TRACE nova.api.openstack     return _iterencode(o, 0)
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.253 TRACE nova.api.openstack
    2014-04-24 11:15:20.259 INFO nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] http://10.0.9.40:8774/v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child returned with HTTP 500
    2014-04-24 11:15:20.259 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=705) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1215
    2014-04-24 11:15:20.260 INFO nova.osapi_compute.wsgi.server [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] 10.0.9.40 "GET /v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child HTTP/1.1" status: 500 len: 335 time: 0.0411668

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1312002

Title:
  nova cell-show <cell name> fails in devstack development environment

Status in OpenStack Compute (Nova):
  New

Bug description:
  I am using devstack development environment with n-cell enabled in localrc.       
  When I call API  to list all cells a cell with name "child" is explored.
  Again when I try to "cell-show" this cell "child" call is failed with an error response 500.
  following is the operations followed - 

   Setting in /etc/nova/nova.conf
           
          [cells]
          name = region
          cell_type = api
          enable = True
           
           


  When API call to get cells -

  REQUEST -          
          curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token:<token>"

  RESPONSE - 
          HTTP/1.1 200 OK
          Content-Type: application/json
          Content-Length: 111
          X-Compute-Request-Id: req-e83c337a-a7d9-47e5-a472-72cd41126f21
          Date: Thu, 24 Apr 2014 04:12:34 GMT
           
          {"cells": [{"username": "guest", "rpc_host": "10.0.9.40", "type": "child", "name": "child", "rpc_port": 5672}]}
           
           
           
  During particular cell show (using cell name as "child")-
           
  REQUEST -
          curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells/child' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: <token>"
           
  RESPONSE - 
          HTTP/1.1 500 Internal Server Error
          Content-Length: 128
          Content-Type: application/json; charset=UTF-8
          X-Compute-Request-Id: req-ba2503bd-3948-4cce-8ee6-543aa597497b
          Date: Thu, 24 Apr 2014 04:10:33 GMT
           
          {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
           

  
  Note: when using cell name as "api" or "region" 404 is returned.
       
       
       
       
  nova-api logs during show call is traced as below -
       
      2014-04-24 11:15:20.219 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token from (pid=705) __call__ /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:603                                                                                                                    
      2014-04-24 11:15:20.220 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role from (pid=705) _remove_auth_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:662                                                                                                                                                          
      2014-04-24 11:15:20.232 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache from (pid=705) _cache_put /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:1121                                                                                                                    
      2014-04-24 11:15:20.233 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: 6cf0c59310fb4b189fc157b19d0e1026 with project_id : 2c3857a83f454b7cb073891ef47acd11 and roles: heat_stack_owner,admin  from (pid=705) _build_user_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:910                                                                                                                                              
      2014-04-24 11:15:20.238 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Matched GET /2c3857a83f454b7cb073891ef47acd11/os-cells/child from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
      2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Route path: '/{project_id}/os-cells/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102                                                                                                                                                
      2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>, 'project_id': u'2c3857a83f454b7cb073891ef47acd11', 'id': u'child'} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103                                                                                                                      
      2014-04-24 11:15:20.240 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Calling method '<bound method Controller.show of <nova.api.openstack.compute.contrib.cells.Controller object at 0x3789c10>>' (Content-type='None', Accept='application/json') from (pid=705) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:945                                                                                                                
      2014-04-24 11:15:20.253 ERROR nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Caught error: Circular reference detected
      Traceback (most recent call last):
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
          incoming.message))
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
          self._send_reply(conn, reply, failure, log_failure=log_failure)
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
          conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
          _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
          return json.dumps(value, default=default, **kwargs)
       
        File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
          **kw).encode(obj)
       
        File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
          chunks = self.iterencode(o, _one_shot=True)
       
        File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
          return _iterencode(o, 0)
       
      ValueError: Circular reference detected
       
      2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return req.get_response(self.application)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     application, catch_exc_info=False)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 616, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.app(env, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     response = self.app(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     content_type, body, accept)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return method(req=request, **action_args)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/common.py", line 602, in inner
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return function(*args, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/cells.py", line 264, in show
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     cell = self.cells_rpcapi.cell_get(context, id)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/cells/rpcapi.py", line 450, in cell_get
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     wait_for_reply=True, timeout=timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     timeout=timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self._send(target, ctxt, message, wait_for_reply, timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     raise result
      2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
      2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     incoming.message))
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     self._send_reply(conn, reply, failure, log_failure=log_failure)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return json.dumps(value, default=default, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     **kw).encode(obj)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     chunks = self.iterencode(o, _one_shot=True)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return _iterencode(o, 0)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.259 INFO nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] http://10.0.9.40:8774/v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child returned with HTTP 500
      2014-04-24 11:15:20.259 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=705) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1215
      2014-04-24 11:15:20.260 INFO nova.osapi_compute.wsgi.server [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] 10.0.9.40 "GET /v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child HTTP/1.1" status: 500 len: 335 time: 0.0411668

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1312002/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next