yahoo-eng-team team mailing list archive

[zhangwenjian <wenjian.zhang@xxxxxxxxx>]

Public bug reported:

When I need to restart my message broker, qpidd, the rpc call from nova-compute may be discrupted and blocked forever. 
Sometimes in service_update and sometimes in periodic_tasks. The blocked call trace of the greenthread is here:


Feb 06 09:28:50 computer01 nova-compute[21244]: ------                        Green Thread                        ------
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:78 in _inner
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.f(*self.args, **self.kw)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:115 in _report_stat
Feb 06 09:28:50 computer01 nova-compute[21244]: `service.service_ref, state_catalog)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/conductor/api.py:218 in service_update
Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._manager.service_update(context, service, values)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:330 in service_update
Feb 06 09:28:50 computer01 nova-compute[21244]: `service=service_p, values=values)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
Feb 06 09:28:50 computer01 nova-compute[21244]: `wait_for_reply=True, timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
Feb 06 09:28:50 computer01 nova-compute[21244]: `timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in send
Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._send(target, ctxt, message, wait_for_reply, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in _send
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._waiter.wait(msg_id, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `reply, ending = self._poll_connection(msg_id, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in _poll_
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.conn.consume(limit=1, timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in consume
Feb 06 09:28:50 computer01 nova-compute[21244]: `six.next(it)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in itercon
Feb 06 09:28:50 computer01 nova-compute[21244]: `yield self.ensure(_error_callback, _consume)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in ensure
Feb 06 09:28:50 computer01 nova-compute[21244]: `return method(*args, **kwargs)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in _consum
Feb 06 09:28:50 computer01 nova-compute[21244]: `nxt_receiver = self.session.next_receiver(timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: <string>:6 in next_receiver
Feb 06 09:28:50 computer01 nova-compute[21244]: (source not found)
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
Feb 06 09:28:50 computer01 nova-compute[21244]: `if self._ecwait(lambda: self.incoming, timeout):`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._ewait(lambda: self.closed or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self.connection._ewait(lambda: self.error or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._wait(lambda: self.error or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._waiter.wait(predicate, timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.condition.wait(timeout - passed)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `sw.wait(timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `ready, _, _ = select([self], [], [], timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
Feb 06 09:28:50 computer01 nova-compute[21244]: `return hub.switch()`
Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
Feb 06 09:28:50 computer01 nova-compute[21244]: `return self.greenlet.switch()`


Feb 05 15:46:29 computer02 nova-compute[20264]: ------                        Green Thread                        ------
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/greenthread.py:194 in main
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = function(*args, **kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:125 in _inner
Feb 05 15:46:29 computer02 nova-compute[20264]: `idle = self.f(*self.args, **self.kw)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/service.py:308 in periodic_tasks
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/manager.py:81 in periodic_tasks
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.run_periodic_tasks(context, raise_on_error=raise_on_error)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:182 in run_p
Feb 05 15:46:29 computer02 nova-compute[20264]: `task(self, context)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/compute/manager.py:5322 in _sync_power_states
Feb 05 15:46:29 computer02 nova-compute[20264]: `use_slave=True)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/objects/base.py:110 in wrapper
Feb 05 15:46:29 computer02 nova-compute[20264]: `args, kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:425 in object_class_action
Feb 05 15:46:29 computer02 nova-compute[20264]: `objver=objver, args=args, kwargs=kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
Feb 05 15:46:29 computer02 nova-compute[20264]: `wait_for_reply=True, timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
Feb 05 15:46:29 computer02 nova-compute[20264]: `timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in send
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self._send(target, ctxt, message, wait_for_reply, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in _send
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._waiter.wait(msg_id, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `reply, ending = self._poll_connection(msg_id, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in _poll_
Feb 05 15:46:29 computer02 nova-compute[20264]: `self.conn.consume(limit=1, timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in consume
Feb 05 15:46:29 computer02 nova-compute[20264]: `six.next(it)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in itercon
Feb 05 15:46:29 computer02 nova-compute[20264]: `yield self.ensure(_error_callback, _consume)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in ensure
Feb 05 15:46:29 computer02 nova-compute[20264]: `return method(*args, **kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in _consum
Feb 05 15:46:29 computer02 nova-compute[20264]: `nxt_receiver = self.session.next_receiver(timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: <string>:6 in next_receiver
Feb 05 15:46:29 computer02 nova-compute[20264]: (source not found)
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
Feb 05 15:46:29 computer02 nova-compute[20264]: `if self._ecwait(lambda: self.incoming, timeout):`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._ewait(lambda: self.closed or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self.connection._ewait(lambda: self.error or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._wait(lambda: self.error or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self._waiter.wait(predicate, timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `self.condition.wait(timeout - passed)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `sw.wait(timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `ready, _, _ = select([self], [], [], timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
Feb 05 15:46:29 computer02 nova-compute[20264]: `return hub.switch()`
Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.greenlet.switch()`

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1418790

Title:
  rpc call is deadlocked when message broker is restarted

Status in OpenStack Compute (Nova):
  New

Bug description:
  When I need to restart my message broker, qpidd, the rpc call from nova-compute may be discrupted and blocked forever. 
  Sometimes in service_update and sometimes in periodic_tasks. The blocked call trace of the greenthread is here:

  
  Feb 06 09:28:50 computer01 nova-compute[21244]: ------                        Green Thread                        ------
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:78 in _inner
  Feb 06 09:28:50 computer01 nova-compute[21244]: `self.f(*self.args, **self.kw)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:115 in _report_stat
  Feb 06 09:28:50 computer01 nova-compute[21244]: `service.service_ref, state_catalog)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/conductor/api.py:218 in service_update
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._manager.service_update(context, service, values)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:330 in service_update
  Feb 06 09:28:50 computer01 nova-compute[21244]: `service=service_p, values=values)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
  Feb 06 09:28:50 computer01 nova-compute[21244]: `wait_for_reply=True, timeout=timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
  Feb 06 09:28:50 computer01 nova-compute[21244]: `timeout=timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in send
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._send(target, ctxt, message, wait_for_reply, timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in _send
  Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._waiter.wait(msg_id, timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in wait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `reply, ending = self._poll_connection(msg_id, timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in _poll_
  Feb 06 09:28:50 computer01 nova-compute[21244]: `self.conn.consume(limit=1, timeout=timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in consume
  Feb 06 09:28:50 computer01 nova-compute[21244]: `six.next(it)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in itercon
  Feb 06 09:28:50 computer01 nova-compute[21244]: `yield self.ensure(_error_callback, _consume)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in ensure
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return method(*args, **kwargs)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in _consum
  Feb 06 09:28:50 computer01 nova-compute[21244]: `nxt_receiver = self.session.next_receiver(timeout=timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: <string>:6 in next_receiver
  Feb 06 09:28:50 computer01 nova-compute[21244]: (source not found)
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
  Feb 06 09:28:50 computer01 nova-compute[21244]: `if self._ecwait(lambda: self.incoming, timeout):`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._ewait(lambda: self.closed or predicate(), timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self.connection._ewait(lambda: self.error or predicate(), timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._wait(lambda: self.error or predicate(), timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._waiter.wait(predicate, timeout=timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `self.condition.wait(timeout - passed)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `sw.wait(timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
  Feb 06 09:28:50 computer01 nova-compute[21244]: `ready, _, _ = select([self], [], [], timeout)`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return hub.switch()`
  Feb 06 09:28:50 computer01 nova-compute[21244]: /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
  Feb 06 09:28:50 computer01 nova-compute[21244]: `return self.greenlet.switch()`

  
  Feb 05 15:46:29 computer02 nova-compute[20264]: ------                        Green Thread                        ------
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/greenthread.py:194 in main
  Feb 05 15:46:29 computer02 nova-compute[20264]: `result = function(*args, **kwargs)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:125 in _inner
  Feb 05 15:46:29 computer02 nova-compute[20264]: `idle = self.f(*self.args, **self.kw)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/service.py:308 in periodic_tasks
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/manager.py:81 in periodic_tasks
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.run_periodic_tasks(context, raise_on_error=raise_on_error)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:182 in run_p
  Feb 05 15:46:29 computer02 nova-compute[20264]: `task(self, context)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/compute/manager.py:5322 in _sync_power_states
  Feb 05 15:46:29 computer02 nova-compute[20264]: `use_slave=True)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/objects/base.py:110 in wrapper
  Feb 05 15:46:29 computer02 nova-compute[20264]: `args, kwargs)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:425 in object_class_action
  Feb 05 15:46:29 computer02 nova-compute[20264]: `objver=objver, args=args, kwargs=kwargs)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
  Feb 05 15:46:29 computer02 nova-compute[20264]: `wait_for_reply=True, timeout=timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
  Feb 05 15:46:29 computer02 nova-compute[20264]: `timeout=timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in send
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return self._send(target, ctxt, message, wait_for_reply, timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in _send
  Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._waiter.wait(msg_id, timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in wait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `reply, ending = self._poll_connection(msg_id, timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in _poll_
  Feb 05 15:46:29 computer02 nova-compute[20264]: `self.conn.consume(limit=1, timeout=timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in consume
  Feb 05 15:46:29 computer02 nova-compute[20264]: `six.next(it)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in itercon
  Feb 05 15:46:29 computer02 nova-compute[20264]: `yield self.ensure(_error_callback, _consume)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in ensure
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return method(*args, **kwargs)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in _consum
  Feb 05 15:46:29 computer02 nova-compute[20264]: `nxt_receiver = self.session.next_receiver(timeout=timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: <string>:6 in next_receiver
  Feb 05 15:46:29 computer02 nova-compute[20264]: (source not found)
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
  Feb 05 15:46:29 computer02 nova-compute[20264]: `if self._ecwait(lambda: self.incoming, timeout):`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._ewait(lambda: self.closed or predicate(), timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self.connection._ewait(lambda: self.error or predicate(), timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._wait(lambda: self.error or predicate(), timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return self._waiter.wait(predicate, timeout=timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `self.condition.wait(timeout - passed)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `sw.wait(timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
  Feb 05 15:46:29 computer02 nova-compute[20264]: `ready, _, _ = select([self], [], [], timeout)`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return hub.switch()`
  Feb 05 15:46:29 computer02 nova-compute[20264]: /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
  Feb 05 15:46:29 computer02 nova-compute[20264]: `return self.greenlet.switch()`

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1418790/+subscriptions