yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** No longer affects: nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1371084

Title:
  nova-scheduler high cpu usage

Status in Messaging API for OpenStack:
  Incomplete

Bug description:
  For no particular reason nova-scheduler cpu utilization can jump to
  100%. I was unable to find any pattern and reason why this is
  happening. We've small cluster 1 cloud controller and 7 node
  controllers. Except high cpu usage nothing bad happens, we're able to
  create/delete instances, after nova-scheduler restart everything goes
  back to normal state.

  I was able to strace 2 processes while nova-scheduler was using 100%
  cpu.

  1st process is in loop and it prints:
  <12>2014-09-16 00:02:21.501 5668 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 12.322771 sec\0"

  2nd processes is in loop as well and it's repeating:
  epoll_ctl(6, EPOLL_CTL_ADD, 3, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=3, u64=40095890530107395}}) = 0
  epoll_wait(6, {{EPOLLOUT, {u32=3, u64=40095890530107395}}}, 1023, 878) = 1
  epoll_ctl(6, EPOLL_CTL_DEL, 3, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLMSG|EPOLLHUP|0x2485d020, {u32=32708, u64=23083065509183428}}) = 0
  sendto(3, "<14>2014-09-16 00:44:19.272 5673 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.3.128.254:5672\0", 125, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)

  Other processes doesn't have any issues with AMQP server just only
  nova-scheduler.

  We're using Debian, kernel 3.14-1-amd64, nova-scheduler 2014.1.2-1.

To manage notifications about this bug go to:
https://bugs.launchpad.net/oslo.messaging/+bug/1371084/+subscriptions