yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Also affects: neutron
   Importance: Undecided
       Status: New

** Also affects: neutron/icehouse
   Importance: Undecided
       Status: New

** Changed in: neutron/icehouse
       Status: New => In Progress

** Changed in: neutron
       Status: New => Fix Committed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1303890

Title:
  Uncaught qpid error can break a consumer

Status in OpenStack Neutron (virtual network service):
  Fix Committed
Status in neutron icehouse series:
  In Progress
Status in Oslo - a Library of Common OpenStack Code:
  Fix Committed
Status in oslo havana series:
  Fix Committed
Status in oslo icehouse series:
  Fix Committed
Status in Messaging API for OpenStack:
  Fix Committed

Bug description:
  The following exception was originally observed against the old rpc
  code, but the same problem exists in oslo.messaging.

   Traceback (most recent call last):
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
       return infunc(*args, **kwargs)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
       self.consume()
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
       it.next()
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
       yield self.ensure(_error_callback, _consume)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
       return method(*args, **kwargs)
     File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
       nxt_receiver = self.session.next_receiver(timeout=timeout)
     File "<string>", line 6, in next_receiver
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
       if self._ecwait(lambda: self.incoming, timeout):
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
       result = self._ewait(lambda: self.closed or predicate(), timeout)
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
       result = self.connection._ewait(lambda: self.error or predicate(), timeout)
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
       self.check_error()
     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
       raise self.error
   InternalError: Traceback (most recent call last):
     File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 667, in write
       self._op_dec.write(*self._seg_dec.read())
     File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
       if self.op.headers is None:
   AttributeError: 'NoneType' object has no attribute 'headers'

  It's possible for something to put the qpid client into a bad state.
  In particular, I have observed a case that will cause
  session.next_receiver() to immediately raise an InternalError.  This
  exception makes it all the way out.  If the eventlet executor is used,
  the forever_retry_uncaught_exceptions() decorator will get hit.  It
  will go back into this code and get the same error, stuck in an
  infinite loop of retrying.

  The connection needs to be reset in this case to recover.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1303890/+subscriptions