yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => havana-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1200069

Title:
  heartbeating is broken with kombu

Status in OpenStack Compute (Nova):
  Fix Released
Status in Oslo - a Library of Common OpenStack Code:
  Fix Committed

Bug description:
  This review:

  https://review.openstack.org/#/c/34949/3

  Added a heartbeat config to kombu... however, kombu requires that you
  call heartbeat_check() periodically to verify that the heartbeats have
  been received.  No such call was added.

  This results in kombu closing the connection every 60 seconds or so
  (the default heartbeat config) and causing a lot of tracebacks in logs
  when trying to consume messages.  kombu ends up raising 'Socket
  closed' and reconnecting:

  2013-07-10 22:01:04.822 13893 ERROR nova.openstack.common.rpc.common [-] Failed
  to consume message from queue: Socket closed
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 589, in ensure
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     return method(*args, **kwargs)
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 669, in _consume
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/kombu/connection.py", line 281, in drain_events
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     return connection.drain_events(**kwargs)
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/amqp/connection.py", line 266, in drain_events
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     chanmap, None, timeout=timeout,
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/amqp/connection.py", line 328, in _wait_multiple
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/amqp/connection.py", line 292, in read_timeout
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     return self.method_reader.read_method()
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common   File "/opt/nova/lib/python2.6/site-packages/amqp/method_framing.py", line 187, in read_method
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common     raise m
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common IOError: Socket closed
  2013-07-10 22:01:04.822 13893 TRACE nova.openstack.common.rpc.common
  2013-07-10 22:01:04.829 13893 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on <foo>:5672

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1200069/+subscriptions