yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1460652] [NEW] nova-conductor infinitely reconnets to rabbit

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Michael Kazakov <1460652@xxxxxxxxxxxxxxxxxx>
Date: Mon, 01 Jun 2015 13:05:07 -0000
Reply-to: Bug 1460652 <1460652@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

1. Exact version of Nova 
ii  nova-api                            1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - API frontend
ii  nova-cert                           1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - certificate management
ii  nova-common                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - common files
ii  nova-conductor                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - conductor service
ii  nova-console                        1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console
ii  nova-consoleauth                    1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console Authenticatorii  nova-novncproxy                     1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - virtual machine scheduler
ii  python-nova                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.17.0.74.g2598714+git201404220131~trusty-0ubuntu1 all          client library for OpenStack Compute API

rabbit configuration in nova.conf:

  rabbit_hosts = m610-2:5672, m610-1:5672
  rabbit_ha_queues =  true


2. Relevant log files:
/var/log/nova/nova-conductor.log

 exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit 
2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
2015-06-01 08:23:56.486 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-06-01 08:23:56.489 16425 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_bea18a6133c548f099b85b168fddf83c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit 
2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-06-01 08:23:56.704 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-2:5672
2015-06-01 08:23:56.707 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
2015-06-01 08:23:56.709 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit 
2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-1:5672
2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-06-01 08:23:56.713 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit 
2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

/var/log/rabbitmq/rabbit@xxxxxxxxxx


=ERROR REPORT==== 1-Jun-2015::08:56:38 ===
connection <0.300.15>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
            'exchange.declare'}

=ERROR REPORT==== 1-Jun-2015::08:56:38 ===
connection <0.294.15>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
            'exchange.declare'}

=ERROR REPORT==== 1-Jun-2015::08:56:38 ===
connection <0.297.15>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
            'exchange.declare'}

=ERROR REPORT==== 1-Jun-2015::08:56:38 ===
connection <0.303.15>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
            'exchange.declare'}


3. Reproduce steps:
2 controller nodes with rabbitMQ HA cluster and nova services (m610-2, m610-1)
rabbit configuration in nova.conf:
It happens after  crush one of controllers. Restarting of nova-conductor service does not solving this problem.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: messaging

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1460652

Title:
  nova-conductor  infinitely reconnets to rabbit

Status in OpenStack Compute (Nova):
  New

Bug description:
  1. Exact version of Nova 
  ii  nova-api                            1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - API frontend
  ii  nova-cert                           1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - certificate management
  ii  nova-common                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - common files
  ii  nova-conductor                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - conductor service
  ii  nova-console                        1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console
  ii  nova-consoleauth                    1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - Console Authenticatorii  nova-novncproxy                     1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                      1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                         1:2014.1.100+git201410062002~trusty-0ubuntu1         all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.17.0.74.g2598714+git201404220131~trusty-0ubuntu1 all          client library for OpenStack Compute API

  rabbit configuration in nova.conf:

    rabbit_hosts = m610-2:5672, m610-1:5672
    rabbit_ha_queues =  true

  
  2. Relevant log files:
  /var/log/nova/nova-conductor.log

   exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.484 16427 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.486 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
  2015-06-01 08:23:56.486 16427 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.489 16425 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_bea18a6133c548f099b85b168fddf83c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'
  2015-06-01 08:23:56.489 16425 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.491 16425 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.704 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-2:5672
  2015-06-01 08:23:56.707 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on m610-1:5672
  2015-06-01 08:23:56.709 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.709 16429 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-1:5672
  2015-06-01 08:23:56.712 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-06-01 08:23:56.713 16429 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c': Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 729, in _publish
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 361, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     type='direct', **options)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 612, in exchange_declare
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 75, in wait
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in dispatch_method
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 232, in _close
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit     reply_code, reply_text, (class_id, method_id), ChannelError,
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit NotFound: Exchange.declare: (404) NOT_FOUND - no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'
  2015-06-01 08:23:56.713 16429 TRACE oslo.messaging._drivers.impl_rabbit 
  2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on m610-2:5672
  2015-06-01 08:23:56.714 16429 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

  /var/log/rabbitmq/rabbit@xxxxxxxxxx

  
  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.300.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.294.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.297.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_bea18a6133c548f099b85b168fddf83c' in vhost '/'",
              'exchange.declare'}

  =ERROR REPORT==== 1-Jun-2015::08:56:38 ===
  connection <0.303.15>, channel 1 - soft error:
  {amqp_error,not_found,
              "no exchange 'reply_7ff0d26e85d94c129c7ae0c2d9fef40c' in vhost '/'",
              'exchange.declare'}

  
  3. Reproduce steps:
  2 controller nodes with rabbitMQ HA cluster and nova services (m610-2, m610-1)
  rabbit configuration in nova.conf:
  It happens after  crush one of controllers. Restarting of nova-conductor service does not solving this problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1460652/+subscriptions
Follow ups

[Bug 1460652] Re: nova-conductor infinitely reconnects to rabbit
From: Davanum Srinivas (DIMS), 2015-07-13
[Bug 1460652] Re: nova-conductor infinitely reconnects to rabbit
From: Mehdi Abaakouk, 2015-06-17
[Bug 1460652] Re: nova-conductor infinitely reconnets to rabbit
From: Davanum Srinivas (DIMS), 2015-06-02
[Bug 1460652] [NEW] nova-conductor infinitely reconnets to rabbit
From: Michael Kazakov, 2015-06-01
References

[Bug 1460652] [NEW] nova-conductor infinitely reconnets to rabbit
From: Michael Kazakov, 2015-06-01