openstack team mailing list archive

["Alex Lyakas" <alex@xxxxxxxxxxxxxxxxx>]

Hi,
I am seeing an issue with the FanoutConsumer: currently each nova service 
declares a FanoutConsumer (in addition to TopicConsumer and DirectConsumer). 
The FanoutConsumer declares a queue on the rabbitmq server, whose name is 
“<topic>_fanout_<UUID>”, for example: 
compute_fanout_e494cb57953b4e6d9dc68c72fb3e26b6.

This queue is marked as “exclusive” and “auto-delete”. “Exclusive” means 
that the queue can be used by a single Connection. However, when the 
FanoutConsumer detects disconnection and attempts to reconnect, it creates a 
new BrokerConnection, but uses the same queue name. In most cases, since the 
queue is marked as “auto-delete”, the rabbitmq server deletes the queue 
automatically on disconnection (after some timeout). So when FanoutConsumer 
tries to reconnect, the previous queue is gone. In some cases, however, the 
queue has not been deleted on the server yet, so when FanoutConsumer 
reconnects, there is an exception on the server:
=ERROR REPORT==== 26-Dec-2012::18:44:55 ===
connection <0.747.0>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 
'volume_fanout_dcc7e1a2d71845d29cd53cfab8920289' in vhost '/'",
'queue.declare'}

As a result, the call to queue.declare() that leads to exception on the 
server is stuck and never returns. So the service is not able to receive any 
messages.

Possible ways to fix this:
1) Use non-exclusive queues (but still auto-delete). (Note: exclusive queues 
are also used by DirectConsumers.) I am not sure what is the implication of 
that in nova. I think it should work, because all exclusive queues defined 
by nova have unique names anyways.
2) Use a different queue name each time FanoutConsumer reconnects. For 
DirectConsumer, this still leave the issue not fixed, although with lower 
probability to happen, I believe.

I implemented a quick fix as in 2), and ensured that it resolves the issue.

According to the latest nova source code, the same issue still exists there.

Another note: “exclusive=true” argument is also used when creating Exchanges 
on the publisher side. However, Exchange C’tor does not receive “exclusive” 
parameter at all (looking at kombu sources), so I am not sure what this does 
(latest nova code also has this).

Can anybody of nova rpc developers please comment on how you think this 
issue should be addressed?

Thanks,
Alex.