yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2056756] Re: A source_type=blank instance was unexpectedly scheduled to the ironic node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <2056756@xxxxxxxxxxxxxxxxxx>
Date: Tue, 12 Mar 2024 10:40:56 -0000
Reply-to: Bug 2056756 <2056756@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Ironic nodes are seen exactly as nova-compute libvirt nodes. If you want
to avoid them, you need to use aggregates.


** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056756

Title:
  A source_type=blank instance was unexpectedly scheduled to the ironic
  node

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========

  I execute the following command to boot an instance with
  source_type=blank of root volume, My OpenStack env has many nodes
  including nova and ironic, the instance was unexpectedly scheduled to
  the ironic node, I check the rest resource and find that it's
  exceeded.

  Could anyone give me some advice to avoid it? Thanks a lot.

  nova boot --flavor 10 --block-device
  source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-
  name=share_net test

  Steps to reproduce
  ==================

  Execute the command to boot the instance:
  nova boot --flavor 10 --block-device source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-name=share_net test

  Expected result
  ===============
  None of node was scheduled, and the instance status will be error.

  Actual result
  =============
  The instance was unexpectedly scheduled to ironic node.

  Environment
  ===========
  Wallaby

  
  Logs & Configs
  ==============

  1.nova-scheduler:

  2024-03-11 20:00:03.632 17 INFO nova.scheduler.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting to schedule for instances: ['953d0c4c-b53e-4739-8444-80ac7442f612']^[[00m
  2024-03-11 20:00:03.788 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting with 14 host(s)^[[00m
  2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AvailabilityZoneFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.790 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeCapabilitiesFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ImagePropertiesFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAntiAffinityFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAffinityFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.819 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter NUMATopologyFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.820 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AggregateVolumeTypeFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.822 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter SriovPciFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter GPUFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter VGPUFilter returned 14 host(s)^[[00m
  2024-03-11 20:00:03.824 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filtered [(ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0]^[[00m
  2024-03-11 20:00:03.825 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Weighed [WeighedHost [host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0]]
  2024-03-11 20:00:03.967 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Selected host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0^[[00m

  2.error output of ironic compute node

  2024-03-11 20:00:04.260 43484 INFO nova.compute.claims [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Claim successful on node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8^[[00m
  2024-03-11 20:00:07.937 43484 INFO nova.virt.block_device [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Booting with blank volume at /dev/sda^[[00m
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Instance failed to spawn: nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Traceback (most recent call last):
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2654, in _build_resources
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]     yield resources
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2427, in _build_and_run_instance
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]     accel_info=accel_info)
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]   File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1195, in spawn
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]     self._node_validate(node, instance, network_info)
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]   File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1149, in _node_validate
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612]     'storage': validate_chk.storage})
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
  2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] ^[[00m
  2024-03-11 20:00:14.490 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Terminating instance^[[00m
  2024-03-11 20:00:14.523 43484 WARNING nova.virt.ironic.driver [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Destroy called on non-existing instance 953d0c4c-b53e-4739-8444-80ac7442f612.: nova.exception.InstanceNotFound: Instance 953d0c4c-b53e-4739-8444-80ac7442f612 could not be found.^[[00m
  2024-03-11 20:00:14.524 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.03 seconds to destroy the instance on the hypervisor.^[[00m
  2024-03-11 20:00:14.732 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.21 seconds to detach 1 volumes for instance.

  3.current nova hypervisor-list

  ()[root@busybox-openstack-866fc955bb-6fgv8 /]# nova hypervisor-list
  +--------------------------------------+--------------------------------------+-------+----------+
  | ID                                   | Hypervisor hostname                  | State | Status   |
  +--------------------------------------+--------------------------------------+-------+----------+
  | d7ef4408-2a9a-447b-af8a-1914c819ff82 | node-4.domain.tld                    | up    | disabled |
  | 66f39f29-f921-4566-985a-38d4b8add595 | node-5.domain.tld                    | up    | disabled |
  | 3da2e264-253d-4da7-8f76-807e9df0627d | node-6.domain.tld                    | up    | enabled  |
  | a98df7b6-a97e-4fbb-b45c-4599f4babd54 | a98df7b6-a97e-4fbb-b45c-4599f4babd54 | up    | enabled  |
  | bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | up    | enabled  |
  | ac1761a6-6ffe-4527-9a2f-1042a101cf57 | ac1761a6-6ffe-4527-9a2f-1042a101cf57 | up    | enabled  |
  | 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | up    | enabled  |
  | 19bbf021-76b4-4222-a633-4539a2c70225 | 19bbf021-76b4-4222-a633-4539a2c70225 | up    | enabled  |
  | 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | up    | enabled  |
  | bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | up    | enabled  |
  | 9cf4ee11-6caa-4c76-9322-1312809b378e | 9cf4ee11-6caa-4c76-9322-1312809b378e | up    | enabled  |
  | fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | up    | enabled  |
  | 9d88099a-fb45-4f0b-aded-d1b87236d76f | 9d88099a-fb45-4f0b-aded-d1b87236d76f | up    | enabled  |
  | fdee4953-5e75-47dd-8cbb-755fd746bd46 | fdee4953-5e75-47dd-8cbb-755fd746bd46 | up    | enabled  |
  | 78676b96-a4fb-4dd6-9aaa-1930732aeeda | 78676b96-a4fb-4dd6-9aaa-1930732aeeda | up    | enabled  |
  | 817a882b-8690-43a6-b310-3fb42a4b07ba | 817a882b-8690-43a6-b310-3fb42a4b07ba | up    | enabled  |
  | 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | up    | enabled  |
  | e693c0fc-f714-453b-b027-bd62a08c3f4c | e693c0fc-f714-453b-b027-bd62a08c3f4c | up    | enabled  |
  | 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | up    | enabled  |
  | 89abe9f3-59d3-4e8e-a460-1814cc759b89 | 89abe9f3-59d3-4e8e-a460-1814cc759b89 | up    | enabled  |
  +--------------------------------------+--------------------------------------+-------+----------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056756/+subscriptions
References

[Bug 2056756] [NEW] A source_type=blank instance was unexpectedly scheduled to the ironic node
From: zhou zhong, 2024-03-11