yahoo-eng-team team mailing list archive

[Joe Gordon <1393182@xxxxxxxxxxxxxxxxxx>]

based on the log this may be an oslo messaging issue.

** Also affects: oslo.messaging
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1393182

Title:
  nova compute failed to report health when nova conductor started

Status in Compass - Open Deployment Service:
  Confirmed
Status in OpenStack Compute (Nova):
  New
Status in Messaging API for OpenStack:
  New

Bug description:
  I have an icehouse openstack deployment that includes one controller and 3 computes. the controller was the last to go up and running. when rabbitmq started, nova compute tried to connect to it. As the log shows, it seemed that it finally got connected, but when doing nova service-list, the nova-compute service was still down. I later restarted nova-compute, and this time, nova service-list showed that the nova-compute became UP.
  I still have two other nova compute remain down status. probably if restarted, they would become UP as well. If any more info is needed, let me know.

  
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 622, in ensure
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 702, in _consume
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return self.connection.drain_events(timeout=timeout)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.drain_events(self.connection, **kwargs)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return connection.drain_events(**kwargs)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return self.wait_multi(self.channels.values(), timeout=timeout)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 81, in wait_multi
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(channel, args)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 365, in _close
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     self._x_close_ok()
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 384, in _x_close_ok
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     self._send_method((10, 61))
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 70, in _send_method
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     method_sig, args, content)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 233, in write_method
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     self.dest.write_frame(1, channel, payload)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 125, in write_frame
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce))
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 359, in sendall
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 342, in send
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
  2014-11-14 21:34:43.449 4314 TRACE oslo.messaging._drivers.impl_rabbit
  2014-11-14 21:34:43.450 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672
  2014-11-14 21:34:43.450 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2014-11-14 21:34:44.501 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672
  2014-11-14 21:34:44.933 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672
  2014-11-14 21:34:44.933 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2014-11-14 21:34:45.982 4314 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672

To manage notifications about this bug go to:
https://bugs.launchpad.net/compass/+bug/1393182/+subscriptions