yahoo-eng-team team mailing list archive

[Michael Still <mikal@xxxxxxxxxxx>]

Resolving this one, but please reopen if you disagree. I have filed
https://bugs.launchpad.net/nova/+bug/1154162 to track adding caching to
our calls to "qemu-img info".

** Changed in: nova
   Importance: High => Undecided

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1078811

Title:
  nova-compute instance creation time increases as instance count
  increases

Status in OpenStack Compute (Nova):
  Invalid
Status in OpenStack Manuals:
  New

Bug description:
  I'm seeing some odd behavior with nova-compute taking exponentially
  longer to fully create/spawn instances as the number of instances on
  any given node increases.

  This doesn't seem to be load/CPU utilization related as I'm  still
  fairly idle on that front (my two nova-compute nodes have 40 cores
  each....4x 10 core sockets).

  Occasionally the instance spawn seems to get caught in some audit task
  where nova-compute will run 'qemu-img info' against each of the
  instance disks. This process typically inputs a very large delay
  between the instance spawn AMQP message to nova-compute (see timestamp
  2012-11-13 05:59:54 in the attached logfile), and nova-compute
  generating the XML for the instance to send to libvirt/KVM (see
  timestamp 2012-11-13 06:19:26 in the attached logfile). While in this
  example it took about 20 minutes, this can cause the instances to take
  as much as an hour to spawn, depending on how many instances are
  hosted on that nova-compute node.

  In other occasions (even with a similarly high instance count, and
  sometimes right after an instance that took significantly longer to
  create), there is no delay between the instance creation task start
  (see timestamp 2012-11-13 06:30:09 in the attached logfile) and the
  XML generation for libvirt (see timestamp 2012-11-13 06:30:21 in the
  attached logfile). Likewise, these instances create predictably and
  consistently (usually about 2 or 3 minutes after sending the XML to
  libvirt/KVM on a busy host).

  The inconsistent and sometimes excessively long instance creation time
  is definitely causing me some headaches with automation and general
  expectations to set around performance in our OpenStack environment.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1078811/+subscriptions