yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1460652] Re: nova-conductor infinitely reconnects to rabbit

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mehdi Abaakouk <sileht@xxxxxxxxxx>
Date: Wed, 17 Jun 2015 13:31:02 -0000
Reply-to: Bug 1460652 <1460652@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Restarting a rpc client can lead to a connection starvation on the connection pool on the rpc server side.

Step that lead to this issue:
* The rpc client sends a bunch of message (> of the connection pool size, 30 by default)
* The rpc server receives all this messages and process it (but don't sent yet the reply)
* The rpc client application is restarted
* The rpc server tries to replies to all messages received before the restart
   * here the reply queue doesn't exists anymore
   * for each messages that need to be replies we wait 60 seconds (in case of this is due to a rabbit restart)
* In the meantime the new rpc client try to send message and expected reply, 
   but the rpc server is waiting to old rpc client to come back.
   Here we got a ton of RPC timeout until the rpc server finished to process its replies messages

** Changed in: oslo.messaging
     Assignee: (unassigned) => Mehdi Abaakouk (sileht)

** Changed in: nova
       Status: New => Invalid

** Changed in: oslo.messaging
       Status: New => Confirmed

** Changed in: oslo.messaging
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1460652

Title:
  nova-conductor  infinitely reconnects to rabbit

Status in OpenStack Compute (Nova):
  Invalid
Status in Messaging API for OpenStack:
  Confirmed

Bug description:
  1. Exact version of Nova 
  ii  nova-api                            1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - API frontend
  ii  nova-cert                           1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - certificate management
  ii  nova-common                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - common files
  ii  nova-conductor                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - conductor service
  ii  nova-console                        1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console
  ii  nova-consoleauth                    1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console Authenticatorii  nova-novncproxy                     1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.17.0.74.g2598714+git201404220131~trusty-0ubuntu1 all          client library for OpenStack Compute API

  rabbit configuration in nova.conf:

    rabbit_hosts = m610-2:5672, m610-1:5672
    rabbit_ha_queues =  true

  
  2. Relevant log files:
  /var/log/nova/nova-conductor.log

   exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.486 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
  2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.489 16425 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_bea18a6133c548f099b85b168fddf83c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.704 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-2:5672
  2015-06-01 08:23:56.707 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
  2015-06-01 08:23:56.709 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-1:5672
  2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.713 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

  /var/log/rabbitmq/rabbit@xxxxxxxxxx

  
  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.300.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.294.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.297.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.303.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
              'exchange.declare'}

  
  3. Reproduce steps:
  2 controller nodes with rabbitMQ HA cluster and nova services (m610-2, m610-1)
  rabbit configuration in nova.conf:
  It happens after  crush one of controllers. Restarting of nova-conductor service does not solving this problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1460652/+subscriptions
References

[Bug 1460652] [NEW] nova-conductor infinitely reconnets to rabbit
From: Michael Kazakov, 2015-06-01