yahoo-eng-team team mailing list archive

[Ian Wienand <iwienand@xxxxxxxxxx>]

** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1195097

Title:
  netaddr.ip.IPAddress kills jsonutils.dumps

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I noticed a backtrack in my nova logs

  ---
  Failed storing info cache
   Traceback (most recent call last):
     File "/opt/stack/nova/nova/network/api.py", line 80, in update_instance_cache_with_nw_info
       cache)
     File "/opt/stack/nova/nova/conductor/api.py", line 101, in instance_info_cache_update
       values)
     File "/opt/stack/nova/nova/conductor/rpcapi.py", line 352, in instance_info_cache_update
       return self.call(context, msg, version='1.26')
     File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 125, in call
       result = rpc.call(context, real_topic, msg, timeout)
     File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
       return _get_impl().call(CONF, context, topic, msg, timeout)
     File "/opt/stack/nova/nova/openstack/common/rpc/impl_qpid.py", line 664, in call
       rpc_amqp.get_connection_pool(conf, Connection))
     File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 613, in call
       rv = multicall(conf, context, topic, msg, timeout, connection_pool)
     File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 599, in multicall
       conn.topic_send(topic, rpc_common.serialize_msg(msg), timeout)
     File "/opt/stack/nova/nova/openstack/common/rpc/common.py", line 468, in serialize_msg
       _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
     File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 151, in dumps
       return json.dumps(value, default=default, **kwargs)
     File "/usr/lib64/python2.6/json/__init__.py", line 237, in dumps
       **kw).encode(obj)
     File "/usr/lib64/python2.6/json/encoder.py", line 367, in encode
       chunks = list(self.iterencode(o))
     File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
       for chunk in self._iterencode_dict(o, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
       for chunk in self._iterencode(value, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
       for chunk in self._iterencode_dict(o, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
       for chunk in self._iterencode(value, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
       for chunk in self._iterencode_dict(o, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
       for chunk in self._iterencode(value, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 317, in _iterencode
       for chunk in self._iterencode_default(o, markers):
     File "/usr/lib64/python2.6/json/encoder.py", line 315, in _iterencode
       raise ValueError("Circular reference detected")
   ValueError: Circular reference detected
  ---

  after a fair bit of digging, I resolved this down to the
  "access_ip_v6" and "access_ip_v4" attributes of the instance object.
  Turns out sqlalchemy turns them into netaddr.IPaddress types, which
  don't get serialized properly

  ---
  [stack@rhel conductor]$ python
  Python 2.6.6 (r266:84292, Oct 12 2012, 14:23:48) 
  [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import netaddr
  >>> import gettext
  >>> gettext.install("nova", unicode=1)
  >>> from nova.openstack.common import jsonutils
  >>> n = netaddr.IPAddress("127.0.0.1")
  >>> jsonutils.dumps(n)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 170, in dumps
      return json.dumps(value, default=default, **kwargs)
    File "/usr/lib64/python2.6/json/__init__.py", line 237, in dumps
      **kw).encode(obj)
    File "/usr/lib64/python2.6/json/encoder.py", line 367, in encode
      chunks = list(self.iterencode(o))
    File "/usr/lib64/python2.6/json/encoder.py", line 317, in _iterencode
      for chunk in self._iterencode_default(o, markers):
    File "/usr/lib64/python2.6/json/encoder.py", line 315, in _iterencode
      raise ValueError("Circular reference detected")
  ValueError: Circular reference detected
  >>> import inspect
  >>> inspect.isclass(n)
  False
  >>> print n
  127.0.0.1
  >>> `n`
  "IPAddress('127.0.0.1')"
  >>> 
  ---

  I think the right thing to do is to special-case this?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1195097/+subscriptions