yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #17581
[Bug 1326207] Re: infinitely reschedule instance when boot instance failed
** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => juno-2
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1326207
Title:
infinitely reschedule instance when boot instance failed
Status in OpenStack Compute (Nova):
Fix Released
Bug description:
When I boot an instance that failed by image size bigger than flavor
size, then nova begin to reschedule the instance infinitely.
$ glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
0d366543-54de-48dd-8f49-d62cac6e7c6b Fedora-x86_64-20-20131211.1-sd qcow2 bare 214106112
$ nova boot --flavor 1 --image 0d366543-54de-48dd-8f49-d62cac6e7c6b --nic net-id=c2d921a0-1474-4731-a8b5-c1cc8b9069f4 vm1
+--------------------------------------+------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000002b |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 8a7ru8z7bUQ6 |
| config_drive | |
| created | 2014-06-04T03:24:38Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | cd5a4327-d3ee-45bb-95fa-da7817b3b0b9 |
| image | Fedora-x86_64-20-20131211.1-sda (0d366543-54de-48dd-8f49-d62cac6e7c6b) |
| key_name | - |
| metadata | {} |
| name | vm1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | ac99aeb3fbb24494b3bbb68a8e94cbff |
| updated | 2014-06-04T03:24:38Z |
| user_id | 2c08e17d6fb54dea96cc6d70ba0ee75f |
+--------------------------------------+------------------------------------------------------------------------+
The first error at compute node:
2014-06-04 11:24:58.121 ERROR nova.virt.libvirt.imagebackend [req-45fbcfec-7440-4193-9caa-53b9322e440c admin admin] /opt/stack/data/nova/instances/_base/d1a823
d9305ec292d5f3e13da6755ee8f057b01e virtual size 2147483648 larger than flavor root disk size 1073741824
2014-06-04 11:24:58.121 ERROR nova.compute.manager [req-45fbcfec-7440-4193-9caa-53b9322e440c admin admin] [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] Inst
ance failed to spawn
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] Traceback (most recent call last):
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/compute/manager.py", line 2059, in _build_resources
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] yield resources
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/compute/manager.py", line 1962, in _build_and_run_instance
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] block_device_info=block_device_info)
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/virt/libvirt/driver.py", line 2281, in spawn
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] admin_pass=admin_password)
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/virt/libvirt/driver.py", line 2655, in _create_image
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] project_id=instance['project_id'])
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 192, in cache
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] *args, **kwargs)
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce55413
7d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 386, in create_image
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] self.verify_base_size(base, size)
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 243, in verify_base_size
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] raise exception.FlavorDiskTooSmall()
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] FlavorDiskTooSmall: Flavor's disk is too small for requested image.
2014-06-04 11:24:58.121 TRACE nova.compute.manager [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9]
2014-06-04 11:24:58.122 ERROR root [req-45fbcfec-7440-4193-9caa-53b9322e440c admin admin] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/manager.py", line 2059, in _build_resources\n yield resources\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/manager.py", line 1962, in _build_and_run_instance\n block_device_info=block_device_info)\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/driver.py", line 2281, in spawn\n admin_pass=admin_password)\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/driver.py", line 2655, in _create_image\n project_id=instance[\'project_id\'])\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 192, in cache\n *args, **kwargs)\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 386, in create_image\n self.verify_base_size(base, size)\n', ' File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 243, in verify_base_size\n raise exception.FlavorDiskTooSmall()\n', "FlavorDiskTooSmall: Flavor's disk is too small for requested image.\n"]
The second:
2014-06-04 11:24:58.174 DEBUG nova.compute.utils [req-45fbcfec-7440-4193-9caa-53b9322e440c admin admin] [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] 'NetworkInfo' object has no attribute 'wait' from (pid=26712) notify_about_instance_usage /media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/utils.py:291
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] Traceback (most recent call last):
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/manager.py", line 1966, in _build_and_run_instance
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] network_info=network_info)
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] self.gen.throw(type, value, traceback)
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] File "/media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/manager.py", line 2065, in _build_resources
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] network_info.wait(do_raise=False)
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] AttributeError: 'NetworkInfo' object has no attribute 'wait'
2014-06-04 11:24:58.174 TRACE nova.compute.utils [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9]
2014-06-04 11:24:58.175 DEBUG nova.compute.manager [req-45fbcfec-7440-4193-9caa-53b9322e440c admin admin] [instance: cd5a4327-d3ee-45bb-95fa-da7817b3b0b9] Build of instance cd5a4327-d3ee-45bb-95fa-da7817b3b0b9 was re-scheduled: 'NetworkInfo' object has no attribute 'wait' from (pid=26712) do_build_and_run_instance /media/soulxu1404/da43e419-d334-4dae-9274-04ce554137d9/opt/stack/nova/nova/compute/manager.py:1895
The third is nova reschedule the instance infinitely
For the first error, the nova api check the image size with flavor size, but the image size is actually size that different with virtual size. In libvirt driver check the virtual size.
the image properties as blow:
$ glance show 01efb526-8046-40d0-b276-ba32e570c965
URI: http://cloudcontroller:9292/v1/images/01efb526-8046-40d0-b276-ba32e570c965
Id: 01efb526-8046-40d0-b276-ba32e570c965
Public: Yes
Protected: No
Name: cirros-0.3.2-x86_64-uec
Status: active
Size: 25165824
Disk format: ami
Container format: ami
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Owner: ac99aeb3fbb24494b3bbb68a8e94cbff
Property 'kernel_id': efa41ea7-22f0-4e08-82b7-74eb0368b28f
Property 'ramdisk_id': eb736431-a77b-4994-ad2b-e090c590e57a
Created at: 2014-05-27T07:34:27
Updated at: 2014-05-27T07:34:27
In nova api, it will check the 'Minimum Disk Required'. User should
specific the 'Minimum Disk Required' for image. So I think this isn't
a bug for nova
For second error, that is wrong way to use network_info object, this
should be fixed.
For third error, after change conductor to build new instance, the
retry info is missing. we should fix is also.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1326207/+subscriptions
References
