yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1312002] Re: nova cell-show <cell name> causes ValueError: Circular reference detected

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Markus Zoeller \(markus_z\)" <mzoeller@xxxxxxxxxxxxxxxxxx>
Date: Tue, 05 Jul 2016 09:48:09 -0000
Reply-to: Bug 1312002 <1312002@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY


** Changed in: nova
   Importance: Low => Undecided

** Changed in: nova
       Status: Confirmed => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1312002

Title:
  nova cell-show <cell name> causes ValueError: Circular reference
  detected

Status in OpenStack Compute (nova):
  Expired

Bug description:
  I am using devstack development environment with n-cell enabled in localrc.       
  When I call API  to list all cells a cell with name "child" is explored.
  Again when I try to "cell-show" this cell "child" call is failed with an error response 500.
  following is the operations followed - 

   Setting in /etc/nova/nova.conf
           
          [cells]
          name = region
          cell_type = api
          enable = True
           
           


  When API call to get cells -

  REQUEST -          
          curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token:<token>"

  RESPONSE - 
          HTTP/1.1 200 OK
          Content-Type: application/json
          Content-Length: 111
          X-Compute-Request-Id: req-e83c337a-a7d9-47e5-a472-72cd41126f21
          Date: Thu, 24 Apr 2014 04:12:34 GMT
           
          {"cells": [{"username": "guest", "rpc_host": "10.0.9.40", "type": "child", "name": "child", "rpc_port": 5672}]}
           
           
           
  During particular cell show (using cell name as "child")-
           
  REQUEST -
          curl -i 'http://10.0.9.40:8774/v2/dcab4e3fc2734bad97c43d46bb77d076/os-cells/child' -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: <token>"
           
  RESPONSE - 
          HTTP/1.1 500 Internal Server Error
          Content-Length: 128
          Content-Type: application/json; charset=UTF-8
          X-Compute-Request-Id: req-ba2503bd-3948-4cce-8ee6-543aa597497b
          Date: Thu, 24 Apr 2014 04:10:33 GMT
           
          {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
           

  
  Note: when using cell name as "api" or "region" 404 is returned.
       
       
       
       
  nova-api logs during show call is traced as below -
       
      2014-04-24 11:15:20.219 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token from (pid=705) __call__ /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:603                                                                                                                    
      2014-04-24 11:15:20.220 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role from (pid=705) _remove_auth_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:662                                                                                                                                                          
      2014-04-24 11:15:20.232 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache from (pid=705) _cache_put /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:1121                                                                                                                    
      2014-04-24 11:15:20.233 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: 6cf0c59310fb4b189fc157b19d0e1026 with project_id : 2c3857a83f454b7cb073891ef47acd11 and roles: heat_stack_owner,admin  from (pid=705) _build_user_headers /opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py:910                                                                                                                                              
      2014-04-24 11:15:20.238 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Matched GET /2c3857a83f454b7cb073891ef47acd11/os-cells/child from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
      2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Route path: '/{project_id}/os-cells/:(id)', defaults: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102                                                                                                                                                
      2014-04-24 11:15:20.239 DEBUG routes.middleware [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x38e8490>, 'project_id': u'2c3857a83f454b7cb073891ef47acd11', 'id': u'child'} from (pid=705) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103                                                                                                                      
      2014-04-24 11:15:20.240 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Calling method '<bound method Controller.show of <nova.api.openstack.compute.contrib.cells.Controller object at 0x3789c10>>' (Content-type='None', Accept='application/json') from (pid=705) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:945                                                                                                                
      2014-04-24 11:15:20.253 ERROR nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Caught error: Circular reference detected
      Traceback (most recent call last):
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
          incoming.message))
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
          self._send_reply(conn, reply, failure, log_failure=log_failure)
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
          conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
          _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
       
        File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
          return json.dumps(value, default=default, **kwargs)
       
        File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
          **kw).encode(obj)
       
        File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
          chunks = self.iterencode(o, _one_shot=True)
       
        File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
          return _iterencode(o, 0)
       
      ValueError: Circular reference detected
       
      2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return req.get_response(self.application)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     application, catch_exc_info=False)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 616, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.app(env, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     response = self.app(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return resp(environ, start_response)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     content_type, body, accept)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return method(req=request, **action_args)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/common.py", line 602, in inner
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return function(*args, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/cells.py", line 264, in show
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     cell = self.cells_rpcapi.cell_get(context, id)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/opt/stack/nova/nova/cells/rpcapi.py", line 450, in cell_get
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return cctxt.call(ctxt, 'cell_get', cell_name=cell_name)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     wait_for_reply=True, timeout=timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     timeout=timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return self._send(target, ctxt, message, wait_for_reply, timeout)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     raise result
      2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
      2014-04-24 11:15:20.253 TRACE nova.api.openstack Traceback (most recent call last):
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     incoming.message))
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 67, in reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     self._send_reply(conn, reply, failure, log_failure=log_failure)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 61, in _send_reply
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 317, in serialize_msg
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return json.dumps(value, default=default, **kwargs)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     **kw).encode(obj)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     chunks = self.iterencode(o, _one_shot=True)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack   File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      2014-04-24 11:15:20.253 TRACE nova.api.openstack     return _iterencode(o, 0)
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack ValueError: Circular reference detected
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.253 TRACE nova.api.openstack
      2014-04-24 11:15:20.259 INFO nova.api.openstack [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] http://10.0.9.40:8774/v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child returned with HTTP 500
      2014-04-24 11:15:20.259 DEBUG nova.api.openstack.wsgi [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=705) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1215
      2014-04-24 11:15:20.260 INFO nova.osapi_compute.wsgi.server [req-3ddd31fe-cd31-4c81-b960-81c753566b4a admin demo] 10.0.9.40 "GET /v2/2c3857a83f454b7cb073891ef47acd11/os-cells/child HTTP/1.1" status: 500 len: 335 time: 0.0411668

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1312002/+subscriptions
References

[Bug 1312002] [NEW] nova cell-show <cell name> fails in devstack development environment
From: Mh Raies, 2014-04-24