yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1383715] [NEW] Devstack installation with cells, throws 'circular-reference detected' on nova cell-show command

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vineet Menon <mvineetmenon@xxxxxxxxx>
Date: Tue, 21 Oct 2014 13:26:15 -0000
Reply-to: Bug 1383715 <1383715@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Devstack installtion with local.conf,

[[local|localrc]]
        ADMIN_PASSWORD=secrete
        DATABASE_PASSWORD=secrete
        RABBIT_PASSWORD=secrete
        SERVICE_PASSWORD=secrete
        SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50
        #FIXED_RANGE=172.31.1.0/24
        #FLOATING_RANGE=192.168.20.0/25
        #HOST_IP=10.3.4.5

        ENABLED_SERVICES+=,n-cell

installs a devstack with a child cell (with name child) and a parent
cell. It's possible to create a server in it.

When issued the command nova cell-show child, it throws exception,

ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID: req-
5df5470a-5135-4cca-b301-238b0a9ce7c2)

The relevant stack trace is as follows,

2014-10-21 13:21:34.375 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Circular reference detected
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     self._send_reply(conn, reply, failure, log_failure=log_failure)
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return json.dumps(value, default=default, **kwargs)
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return _iterencode(o, 0)
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher ValueError: Circular reference detected
2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher
2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] Returning exception Circular reference detected to caller
2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply\n    self._send_reply(conn, reply, failure, log_failure=log_failure)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply\n    conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg\n    _MESSAGE_KEY: jsonutils.dumps(raw_msg)}\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps\n    return json.dumps(value, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

** Affects: nova
     Importance: Undecided
     Assignee: Vineet Menon (mvineetmenon)
         Status: New


** Tags: cells

** Changed in: nova
     Assignee: (unassigned) => Vineet Menon (mvineetmenon)

** Description changed:

  Devstack installtion with local.conf,
  
  [[local|localrc]]
-         ADMIN_PASSWORD=secrete
-         DATABASE_PASSWORD=secrete
-         RABBIT_PASSWORD=secrete
-         SERVICE_PASSWORD=secrete
-         SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50
-         #FIXED_RANGE=172.31.1.0/24
-         #FLOATING_RANGE=192.168.20.0/25
-         #HOST_IP=10.3.4.5
+         ADMIN_PASSWORD=secrete
+         DATABASE_PASSWORD=secrete
+         RABBIT_PASSWORD=secrete
+         SERVICE_PASSWORD=secrete
+         SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50
+         #FIXED_RANGE=172.31.1.0/24
+         #FLOATING_RANGE=192.168.20.0/25
+         #HOST_IP=10.3.4.5
  
-         ENABLED_SERVICES+=,n-cell
+         ENABLED_SERVICES+=,n-cell
  
  installs a devstack with a child cell (with name child) and a parent
  cell. It's possible to create a server in it.
  
  When issued the command nova cell-show child, it throws exception,
-  
- ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5df5470a-5135-4cca-b301-238b0a9ce7c2)
+ 
+ ERROR (ClientException): The server has either erred or is incapable of
+ performing the requested operation. (HTTP 500) (Request-ID: req-
+ 5df5470a-5135-4cca-b301-238b0a9ce7c2)
  
  The relevant stack trace is as follows,
  
  2014-10-21 13:21:34.375 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Circular reference detected
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     self._send_reply(conn, reply, failure, log_failure=log_failure)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return json.dumps(value, default=default, **kwargs)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return _iterencode(o, 0)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher ValueError: Circular reference detected
- 2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher 
+ 2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher
  2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] Returning exception Circular reference detected to caller
  2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply\n    self._send_reply(conn, reply, failure, log_failure=log_failure)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply\n    conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg\n    _MESSAGE_KEY: jsonutils.dumps(raw_msg)}\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps\n    return json.dumps(value, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

** Summary changed:

- Devstack installation with cells, breaks on nova cell-show command
+ Devstack installation with cells, throws 'circular-reference detected' on nova cell-show command

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1383715

Title:
  Devstack installation with cells, throws 'circular-reference detected'
  on nova cell-show command

Status in OpenStack Compute (Nova):
  New

Bug description:
  Devstack installtion with local.conf,

  [[local|localrc]]
          ADMIN_PASSWORD=secrete
          DATABASE_PASSWORD=secrete
          RABBIT_PASSWORD=secrete
          SERVICE_PASSWORD=secrete
          SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50
          #FIXED_RANGE=172.31.1.0/24
          #FLOATING_RANGE=192.168.20.0/25
          #HOST_IP=10.3.4.5

          ENABLED_SERVICES+=,n-cell

  installs a devstack with a child cell (with name child) and a parent
  cell. It's possible to create a server in it.

  When issued the command nova cell-show child, it throws exception,

  ERROR (ClientException): The server has either erred or is incapable
  of performing the requested operation. (HTTP 500) (Request-ID: req-
  5df5470a-5135-4cca-b301-238b0a9ce7c2)

  The relevant stack trace is as follows,

  2014-10-21 13:21:34.375 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Circular reference detected
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     self._send_reply(conn, reply, failure, log_failure=log_failure)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return json.dumps(value, default=default, **kwargs)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher     return _iterencode(o, 0)
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher ValueError: Circular reference detected
  2014-10-21 13:21:34.375 TRACE oslo.messaging.rpc.dispatcher
  2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] Returning exception Circular reference detected to caller
  2014-10-21 13:21:34.377 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply\n    self._send_reply(conn, reply, failure, log_failure=log_failure)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 62, in _send_reply\n    conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 302, in serialize_msg\n    _MESSAGE_KEY: jsonutils.dumps(raw_msg)}\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 172, in dumps\n    return json.dumps(value, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1383715/+subscriptions
Follow ups

[Bug 1383715] [NEW] Devstack installation with cells, throws 'circular-reference detected' on nova cell-show command
From: Vineet Menon, 2014-10-21
References

[Bug 1383715] [NEW] Devstack installation with cells, throws 'circular-reference detected' on nova cell-show command
From: Vineet Menon, 2014-10-21