yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** Changed in: nova
       Status: Triaged => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1226170

Title:
  bm node instance provisioning delay with 45 nodes

Status in OpenStack Compute (Nova):
  Won't Fix

Bug description:
  Nova - 1:2013.1

  While doing scale test, within a batch of 42 cartridges, few of them get provisioned in ~7 mins and few takes 35 mins.
  There is totally 10 instances which spans between 30-36 minutes of provisioning time.

  nova-compute log snippet of the node which is taking 36 minutes [the
  highest time in the batch].

  In the first minute, the Claim is successful.
  The next 30 minutes is wait time with the message "During sync_power_state the instance has a pending task. Skip" and eventually the provisioning gets completed.

  Per blueprint, https://blueprints.launchpad.net/nova/+spec/improve-
  baremetal-pxe-deploy

  the current is nova-baremetal-deploy-helper mounts the baremetal node's disks via iSCSI, fdisks the partitions, and dd's the updated glance image over iSCSI
  while the desired approach is ramdisk fdisks the local disks, pulls specified image from glance, writes to local disk, and reboot into it

  Is the current approach causing this performance bottle neck? Is there
  any parameter which can be tuned to better the performance?

  Line 388: 2013-09-06 12:09:12.843 AUDIT nova.compute.manager [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Starting instance...
  Line 801: 2013-09-06 12:10:15.230 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Attempting claim: memory 2048 MB, disk 30 GB, VCPUs 1
  Line 802: 2013-09-06 12:10:15.232 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Total Memory: 2048 MB, used: 512 MB
  Line 803: 2013-09-06 12:10:15.233 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Memory limit: 3072 MB, free: 2560 MB
  Line 804: 2013-09-06 12:10:15.235 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Total Disk: 80 GB, used: 0 GB
  Line 805: 2013-09-06 12:10:15.236 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Disk limit not specified, defaulting to unlimited
  Line 806: 2013-09-06 12:10:15.238 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Total CPU: 1 VCPUs, used: 0 VCPUs
  Line 807: 2013-09-06 12:10:15.240 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] CPU limit not specified, defaulting to unlimited
  Line 808: 2013-09-06 12:10:15.241 AUDIT nova.compute.claims [req-894f8127-4b15-4046-9919-fbd0123f3555 93aabe9ff2064688bdd070f16e6de768 40c74bd952f8470991117d54f0c03f0f] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] Claim successful
  Line 1047: 2013-09-06 12:12:14.514 28732 INFO nova.compute.manager [-] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] During sync_power_state the instance has a pending task. Skip.
  Line 1354: 2013-09-06 12:22:23.266 28732 INFO nova.compute.manager [-] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] During sync_power_state the instance has a pending task. Skip.
  Line 1405: 2013-09-06 12:32:30.858 28732 INFO nova.compute.manager [-] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] During sync_power_state the instance has a pending task. Skip.
  Line 1486: 2013-09-06 12:42:38.456 28732 INFO nova.compute.manager [-] [instance: 15047e3b-b4c8-4b45-906e-5c2e4650a9e2] During sync_power_state the instance has a pending task. Skip.
  Line 1490: 2013-09-06 12:43:58.078 28732 INFO nova.virt.baremetal.pxe [-] PXE deploy started for instance 15047e3b-b4c8-4b45-906e-5c2e4650a9e2
  Line 1493: 2013-09-06 12:44:41.641 28732 INFO nova.virt.baremetal.pxe [-] PXE deploy completed for instance 15047e3b-b4c8-4b45-906e-5c2e4650a9e2

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1226170/+subscriptions