yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1320269] Re: Scheduler fails confirm resize of instances when there is enough space to complete resize

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pushkar Umaranikar <pushkar.umaranikar@xxxxxxxxx>
Date: Mon, 30 Nov 2015 18:18:28 -0000
Reply-to: Bug 1320269 <1320269@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I tried resizing of instances with preallocation strategy for nano and micro flavors and tried same for few instances of tiny flavor.
I could not reproduce bug.
At the same time, I am not able to check with other flavors due to hardware resources restriction I have. But I don't think we need API change here, as it is not reproducible with basic flavors. I think we can close this bug. Please feel free to reopen if required.

** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1320269

Title:
  Scheduler fails confirm resize of instances when there is enough space
  to complete resize

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  I configured preallocation on my setup and launched 10 instances. 
  8 of the 10 instances have enough space to resize 
  I started to resize all instances 1 at a time but did not confirm the resize
  when I got to the 8th instance the scheduler would not allow resize due to disk space. 
  when I tried confirming resize of the instances with VERIFY_RESIZE status we fail on not enough disk space on all of them. 

  [root@orange-vdsf instances(keystone_admin)]# nova list 
  +--------------------------------------+--------------------------------------------+---------------+------------+-------------+---------------------------+
  | ID                                   | Name                                       | Status        | Task State | Power State | Networks                  |
  +--------------------------------------+--------------------------------------------+---------------+------------+-------------+---------------------------+
  | 1a98eefe-ba41-49b7-931a-ebe54796e343 | dafna-1a98eefe-ba41-49b7-931a-ebe54796e343 | ERROR         | -          | Running     | novanetwork=192.168.32.21 |
  | 1e237fb7-6787-4362-8ae3-e9668cdd3b50 | dafna-1e237fb7-6787-4362-8ae3-e9668cdd3b50 | ACTIVE        | -          | Running     | novanetwork=192.168.32.13 |
  | 5f046400-7a8f-4a66-a1b2-5ac87f442e06 | dafna-5f046400-7a8f-4a66-a1b2-5ac87f442e06 | ERROR         | -          | Running     | novanetwork=192.168.32.3  |
  | 5f87fd7e-4f85-4c41-ab12-3bf680160281 | dafna-5f87fd7e-4f85-4c41-ab12-3bf680160281 | ERROR         | -          | Running     | novanetwork=192.168.32.15 |
  | 608eef43-c9ac-48e0-8482-e9275c123268 | dafna-608eef43-c9ac-48e0-8482-e9275c123268 | ERROR         | -          | Running     | novanetwork=192.168.32.6  |
  | 7610109e-b631-40a5-9fa0-1bf150560503 | dafna-7610109e-b631-40a5-9fa0-1bf150560503 | ERROR         | -          | Running     | novanetwork=192.168.32.2  |
  | 7dfc892f-65ef-4e5b-b9bf-35d74a37bb27 | dafna-7dfc892f-65ef-4e5b-b9bf-35d74a37bb27 | ERROR         | -          | Running     | novanetwork=192.168.32.22 |
  | a4aefac9-feb8-40d7-9228-1df6a02d9e13 | dafna-a4aefac9-feb8-40d7-9228-1df6a02d9e13 | VERIFY_RESIZE | -          | Running     | novanetwork=192.168.32.18 |
  | ac4fb242-3fa4-4424-9410-4de61d260e0a | dafna-ac4fb242-3fa4-4424-9410-4de61d260e0a | ERROR         | -          | Running     | novanetwork=192.168.32.12 |
  | d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 | dafna-d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 | ERROR         | -          | Running     | novanetwork=192.168.32.14 |
  +--------------------------------------+--------------------------------------------+---------------+------------+-------------+---------------------------+
  [root@orange-vdsf instances(keystone_admin)]# nova list --host orange-vdsf.qa.lab.tlv.redhat.com
  +--------------------------------------+--------------------------------------------+--------+------------+-------------+---------------------------+


  
  [root@orange-vdsf instances(keystone_admin)]# egrep d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 /var/log/nova/*
  /var/log/nova/nova-api.log:2014-05-16 17:38:45.092 15231 INFO nova.osapi_compute.wsgi.server [req-fe2167ee-705a-4563-a120-ddbd1abfc968 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1923 time: 0.3389881
  /var/log/nova/nova-api.log:2014-05-16 17:38:48.288 15231 INFO nova.osapi_compute.wsgi.server [req-3e8587c1-607f-436c-b351-b1c29576b815 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1923 time: 0.3205199
  /var/log/nova/nova-api.log:2014-05-16 17:38:51.077 15231 INFO nova.osapi_compute.wsgi.server [req-6901a68e-cb70-40a0-bf34-5efbaef492df c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1940 time: 0.4703982
  /var/log/nova/nova-api.log:2014-05-16 17:40:18.526 15231 INFO nova.osapi_compute.wsgi.server [req-5828f84c-959d-45bd-8d39-226f321b11b2 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1940 time: 0.0601959
  /var/log/nova/nova-api.log:2014-05-16 17:40:22.696 15231 INFO nova.osapi_compute.wsgi.server [req-4d10f49f-23c5-47cc-beb4-82a11070c794 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1940 time: 0.0649240
  /var/log/nova/nova-api.log:2014-05-16 17:40:22.972 15231 INFO nova.osapi_compute.wsgi.server [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "POST /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98/action HTTP/1.1" status: 202 len: 185 time: 0.2568388
  /var/log/nova/nova-api.log:2014-05-16 17:40:26.715 15231 INFO nova.osapi_compute.wsgi.server [req-9074bf34-edf8-4331-b26d-b50967d778de c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 1954 time: 0.4957981
  /var/log/nova/nova-api.log:2014-05-16 17:40:39.094 15231 INFO nova.osapi_compute.wsgi.server [req-d3c75e98-1a32-4522-85ee-6ea03cb0817a c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] 10.35.160.73 "GET /v2/4ad766166539403189f2caca1ba306aa/servers/d1e28a92-3ba2-402e-a798-ff8ff9a6bd98 HTTP/1.1" status: 200 len: 2021 time: 0.0873950
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.835 15163 AUDIT nova.compute.manager [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Starting instance...
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.994 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.995 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total memory: 15888 MB, used: 4096.00 MB
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.995 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] memory limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.995 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total disk: 59 GB, used: 34.00 GB
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.995 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] disk limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.995 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total CPUs: 4 VCPUs, used: 7.00 VCPUs
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.996 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] CPUs limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:38:41.996 15163 AUDIT nova.compute.claims [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Claim successful
  /var/log/nova/nova-compute.log:2014-05-16 17:38:43.804 15163 INFO nova.virt.libvirt.driver [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Creating image
  /var/log/nova/nova-compute.log:2014-05-16 17:38:46.286 15163 INFO nova.virt.libvirt.firewall [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Called setup_basic_filtering in nwfilter
  /var/log/nova/nova-compute.log:2014-05-16 17:38:46.286 15163 INFO nova.virt.libvirt.firewall [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Ensuring static filters
  /var/log/nova/nova-compute.log:2014-05-16 17:38:48.534 15163 INFO nova.virt.libvirt.driver [-] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Instance spawned successfully.
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.086 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Attempting claim: memory 512 MB, disk 10 GB, VCPUs 1
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.087 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total memory: 15888 MB, used: 5120.00 MB
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.087 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] memory limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.087 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total disk: 59 GB, used: 45.00 GB
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.087 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] disk limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.087 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Total CPUs: 4 VCPUs, used: 9.00 VCPUs
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.088 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] CPUs limit not specified, defaulting to unlimited
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.088 15163 AUDIT nova.compute.claims [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Claim successful
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.105 15163 AUDIT nova.compute.resource_tracker [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] Updating from migration d1e28a92-3ba2-402e-a798-ff8ff9a6bd98
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.139 15163 AUDIT nova.compute.manager [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Migrating
  /var/log/nova/nova-compute.log:2014-05-16 17:40:23.784 15163 INFO nova.virt.libvirt.driver [-] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Instance destroyed successfully.
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.600 15163 INFO nova.virt.libvirt.driver [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Creating image
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 ERROR nova.compute.manager [req-5007a7d1-e7c5-41cc-b660-a638d25367cf c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Setting instance vm_state to ERROR
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 TRACE nova.compute.manager [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98] Traceback (most recent call last):
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 TRACE nova.compute.manager [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3559, in finish_resize
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 TRACE nova.compute.manager [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98]     disk_info, image)
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 TRACE nova.compute.manager [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3530, in _finish_resize
  /var/log/nova/nova-compute.log:2014-05-16 17:40:39.652 15163 TRACE nova.compute.manager [instance: d1e28a92-3ba2-402e-a798-ff8ff9a6bd98]     block_device_info, power_on)
  /var/log/nova/nova-scheduler.log:2014-05-16 17:38:41.565 15217 INFO nova.scheduler.filter_scheduler [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] Attempting to build 4 instance(s) uuids: [u'7610109e-b631-40a5-9fa0-1bf150560503', u'5f046400-7a8f-4a66-a1b2-5ac87f442e06', u'd1e28a92-3ba2-402e-a798-ff8ff9a6bd98', u'1e237fb7-6787-4362-8ae3-e9668cdd3b50']
  /var/log/nova/nova-scheduler.log:2014-05-16 17:38:41.704 15217 INFO nova.scheduler.filter_scheduler [req-53250d81-dde6-44a9-9419-fe313fd8ec2e c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] Choosing host WeighedHost [host: orange-vdsf.qa.lab.tlv.redhat.com, weight: 1.0] for instance d1e28a92-3ba2-402e-a798-ff8ff9a6bd98
  egrep: /var/log/nova/tmp: Is a directory
  [root@orange-vdsf instances(keystone_admin)]#

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1320269/+subscriptions
References

[Bug 1320269] [NEW] schedual fails confirm resize of instances when there is enough space to complete resize
From: Dafna Ron, 2014-05-16