yahoo-eng-team team mailing list archive

[Alan Pevec <1303890@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron/havana
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1303890

Title:
  Uncaught qpid error can break a consumer

Status in OpenStack Neutron (virtual network service):
  Fix Released
Status in neutron havana series:
  Fix Released
Status in neutron icehouse series:
  Fix Committed
Status in The Oslo library incubator:
  Fix Released
Status in oslo-incubator havana series:
  Fix Committed
Status in oslo-incubator icehouse series:
  Fix Committed
Status in Messaging API for OpenStack:
  Fix Released

Bug description:
  The following exception was originally observed against the old rpc
  code, but the same problem exists in oslo.messaging.

   Traceback (most recent call last):
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
       return infunc(*args, **kwargs)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
       self.consume()
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
       it.next()
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
       yield self.ensure(_error_callback, _consume)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
       return method(*args, **kwargs)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
       nxt_receiver = self.session.next_receiver(timeout=timeout)
     File "<string>", line 6, in next_receiver
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
       if self._ecwait(lambda: self.incoming, timeout):
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
       result = self._ewait(lambda: self.closed or predicate(), timeout)
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
       result = self.connection._ewait(lambda: self.error or predicate(), timeout)
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
       self.check_error()
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
       raise self.error
   InternalError: Traceback (most recent call last):
     File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 667, in write
       self._op_dec.write(*self._seg_dec.read())
     File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
       if self.op.headers is None:
   AttributeError: 'NoneType' object has no attribute 'headers'

  It's possible for something to put the qpid client into a bad state.
  In particular, I have observed a case that will cause
  session.next_receiver() to immediately raise an InternalError.  This
  exception makes it all the way out.  If the eventlet executor is used,
  the forever_retry_uncaught_exceptions() decorator will get hit.  It
  will go back into this code and get the same error, stuck in an
  infinite loop of retrying.

  The connection needs to be reset in this case to recover.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1303890/+subscriptions