yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #93698
[Bug 2056756] [NEW] A source_type=blank instance was unexpectedly scheduled to the ironic node
Public bug reported:
Description
===========
I execute the following command to boot an instance with
source_type=blank of root volume, My OpenStack env has many nodes
including nova and ironic, the instance was unexpectedly scheduled to
the ironic node, I check the rest resource and find that it's exceeded.
Could anyone give me some advice to avoid it? Thanks a lot.
nova boot --flavor 10 --block-device
source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-
name=share_net test
Steps to reproduce
==================
Execute the command to boot the instance:
nova boot --flavor 10 --block-device source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-name=share_net test
Expected result
===============
None of node was scheduled, and the instance status will be error.
Actual result
=============
The instance was unexpectedly scheduled to ironic node.
Environment
===========
Wallaby
Logs & Configs
==============
1.nova-scheduler:
2024-03-11 20:00:03.632 17 INFO nova.scheduler.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting to schedule for instances: ['953d0c4c-b53e-4739-8444-80ac7442f612']^[[00m
2024-03-11 20:00:03.788 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting with 14 host(s)^[[00m
2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AvailabilityZoneFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.790 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeCapabilitiesFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ImagePropertiesFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAntiAffinityFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAffinityFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.819 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter NUMATopologyFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.820 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AggregateVolumeTypeFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.822 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter SriovPciFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter GPUFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter VGPUFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.824 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filtered [(ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0]^[[00m
2024-03-11 20:00:03.825 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Weighed [WeighedHost [host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0]]
2024-03-11 20:00:03.967 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Selected host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0^[[00m
2.error output of ironic compute node
2024-03-11 20:00:04.260 43484 INFO nova.compute.claims [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Claim successful on node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8^[[00m
2024-03-11 20:00:07.937 43484 INFO nova.virt.block_device [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Booting with blank volume at /dev/sda^[[00m
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Instance failed to spawn: nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Traceback (most recent call last):
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2654, in _build_resources
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] yield resources
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2427, in _build_and_run_instance
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] accel_info=accel_info)
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1195, in spawn
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] self._node_validate(node, instance, network_info)
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1149, in _node_validate
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] 'storage': validate_chk.storage})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] ^[[00m
2024-03-11 20:00:14.490 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Terminating instance^[[00m
2024-03-11 20:00:14.523 43484 WARNING nova.virt.ironic.driver [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Destroy called on non-existing instance 953d0c4c-b53e-4739-8444-80ac7442f612.: nova.exception.InstanceNotFound: Instance 953d0c4c-b53e-4739-8444-80ac7442f612 could not be found.^[[00m
2024-03-11 20:00:14.524 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.03 seconds to destroy the instance on the hypervisor.^[[00m
2024-03-11 20:00:14.732 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.21 seconds to detach 1 volumes for instance.
3.current nova hypervisor-list
()[root@busybox-openstack-866fc955bb-6fgv8 /]# nova hypervisor-list
+--------------------------------------+--------------------------------------+-------+----------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+--------------------------------------+-------+----------+
| d7ef4408-2a9a-447b-af8a-1914c819ff82 | node-4.domain.tld | up | disabled |
| 66f39f29-f921-4566-985a-38d4b8add595 | node-5.domain.tld | up | disabled |
| 3da2e264-253d-4da7-8f76-807e9df0627d | node-6.domain.tld | up | enabled |
| a98df7b6-a97e-4fbb-b45c-4599f4babd54 | a98df7b6-a97e-4fbb-b45c-4599f4babd54 | up | enabled |
| bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | up | enabled |
| ac1761a6-6ffe-4527-9a2f-1042a101cf57 | ac1761a6-6ffe-4527-9a2f-1042a101cf57 | up | enabled |
| 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | up | enabled |
| 19bbf021-76b4-4222-a633-4539a2c70225 | 19bbf021-76b4-4222-a633-4539a2c70225 | up | enabled |
| 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | up | enabled |
| bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | up | enabled |
| 9cf4ee11-6caa-4c76-9322-1312809b378e | 9cf4ee11-6caa-4c76-9322-1312809b378e | up | enabled |
| fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | up | enabled |
| 9d88099a-fb45-4f0b-aded-d1b87236d76f | 9d88099a-fb45-4f0b-aded-d1b87236d76f | up | enabled |
| fdee4953-5e75-47dd-8cbb-755fd746bd46 | fdee4953-5e75-47dd-8cbb-755fd746bd46 | up | enabled |
| 78676b96-a4fb-4dd6-9aaa-1930732aeeda | 78676b96-a4fb-4dd6-9aaa-1930732aeeda | up | enabled |
| 817a882b-8690-43a6-b310-3fb42a4b07ba | 817a882b-8690-43a6-b310-3fb42a4b07ba | up | enabled |
| 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | up | enabled |
| e693c0fc-f714-453b-b027-bd62a08c3f4c | e693c0fc-f714-453b-b027-bd62a08c3f4c | up | enabled |
| 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | up | enabled |
| 89abe9f3-59d3-4e8e-a460-1814cc759b89 | 89abe9f3-59d3-4e8e-a460-1814cc759b89 | up | enabled |
+--------------------------------------+--------------------------------------+-------+----------+
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056756
Title:
A source_type=blank instance was unexpectedly scheduled to the ironic
node
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
I execute the following command to boot an instance with
source_type=blank of root volume, My OpenStack env has many nodes
including nova and ironic, the instance was unexpectedly scheduled to
the ironic node, I check the rest resource and find that it's
exceeded.
Could anyone give me some advice to avoid it? Thanks a lot.
nova boot --flavor 10 --block-device
source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-
name=share_net test
Steps to reproduce
==================
Execute the command to boot the instance:
nova boot --flavor 10 --block-device source=blank,dest=volume,size=1,bootindex=0,volume_type=hdd --nic net-name=share_net test
Expected result
===============
None of node was scheduled, and the instance status will be error.
Actual result
=============
The instance was unexpectedly scheduled to ironic node.
Environment
===========
Wallaby
Logs & Configs
==============
1.nova-scheduler:
2024-03-11 20:00:03.632 17 INFO nova.scheduler.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting to schedule for instances: ['953d0c4c-b53e-4739-8444-80ac7442f612']^[[00m
2024-03-11 20:00:03.788 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Starting with 14 host(s)^[[00m
2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AvailabilityZoneFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.789 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.790 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ComputeCapabilitiesFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ImagePropertiesFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAntiAffinityFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.791 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter ServerGroupAffinityFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.819 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter NUMATopologyFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.820 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter AggregateVolumeTypeFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.822 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter SriovPciFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter GPUFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.823 17 INFO nova.filters [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filter VGPUFilter returned 14 host(s)^[[00m
2024-03-11 20:00:03.824 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Filtered [(ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0]^[[00m
2024-03-11 20:00:03.825 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Weighed [WeighedHost [host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 19bbf021-76b4-4222-a633-4539a2c70225) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 89abe9f3-59d3-4e8e-a460-1814cc759b89) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 5b0118c4-0aa4-4c96-9919-e6beb4e886f2) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, fdee4953-5e75-47dd-8cbb-755fd746bd46) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 78676b96-a4fb-4dd6-9aaa-1930732aeeda) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, e693c0fc-f714-453b-b027-bd62a08c3f4c) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.0, ac1761a6-6ffe-4527-9a2f-1042a101cf57) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 3e37dd34-d6d4-4726-93cb-ad4e293835f8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 9d88099a-fb45-4f0b-aded-d1b87236d76f) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.1, 817a882b-8690-43a6-b310-3fb42a4b07ba) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0], WeighedHost [host: (ironic.compute.domain.tld.2, 9cf4ee11-6caa-4c76-9322-1312809b378e) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0, weight: 102.0]]
2024-03-11 20:00:03.967 17 INFO nova.scheduler.filter_scheduler [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Selected host: (ironic.compute.domain.tld.2, fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8) ram: 4096MB disk: 30720MB io_ops: 0 instances: 0^[[00m
2.error output of ironic compute node
2024-03-11 20:00:04.260 43484 INFO nova.compute.claims [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Claim successful on node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8^[[00m
2024-03-11 20:00:07.937 43484 INFO nova.virt.block_device [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Booting with blank volume at /dev/sda^[[00m
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Instance failed to spawn: nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Traceback (most recent call last):
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2654, in _build_resources
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] yield resources
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 2427, in _build_and_run_instance
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] accel_info=accel_info)
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1195, in spawn
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] self._node_validate(node, instance, network_info)
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] File "/usr/local/lib/python3.6/site-packages/nova/virt/ironic/driver.py", line 1149, in _node_validate
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] 'storage': validate_chk.storage})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] nova.exception.ValidationError: Ironic node: fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 failed to validate. (deploy: {'reason': "Cannot validate image information for node fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 because one or more parameters are missing from its instance_info and insufficent information is present to boot from a remote volume. Missing are: ['ramdisk', 'kernel', 'image_source']", 'result': False}, power: {'result': True}, storage: {'result': True})
2024-03-11 20:00:14.489 43484 ERROR nova.compute.manager [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] ^[[00m
2024-03-11 20:00:14.490 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Terminating instance^[[00m
2024-03-11 20:00:14.523 43484 WARNING nova.virt.ironic.driver [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] Destroy called on non-existing instance 953d0c4c-b53e-4739-8444-80ac7442f612.: nova.exception.InstanceNotFound: Instance 953d0c4c-b53e-4739-8444-80ac7442f612 could not be found.^[[00m
2024-03-11 20:00:14.524 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.03 seconds to destroy the instance on the hypervisor.^[[00m
2024-03-11 20:00:14.732 43484 INFO nova.compute.manager [req-4a81e1fd-9acd-43cf-9f53-f5ddb7155acb f2f7c29c86034f0a85e60549601fe5b5 b00eb18beb7647dba928b26485606784 - default default] [instance: 953d0c4c-b53e-4739-8444-80ac7442f612] Took 0.21 seconds to detach 1 volumes for instance.
3.current nova hypervisor-list
()[root@busybox-openstack-866fc955bb-6fgv8 /]# nova hypervisor-list
+--------------------------------------+--------------------------------------+-------+----------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+--------------------------------------+-------+----------+
| d7ef4408-2a9a-447b-af8a-1914c819ff82 | node-4.domain.tld | up | disabled |
| 66f39f29-f921-4566-985a-38d4b8add595 | node-5.domain.tld | up | disabled |
| 3da2e264-253d-4da7-8f76-807e9df0627d | node-6.domain.tld | up | enabled |
| a98df7b6-a97e-4fbb-b45c-4599f4babd54 | a98df7b6-a97e-4fbb-b45c-4599f4babd54 | up | enabled |
| bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | bc9943f8-8d0f-4eff-aaa6-250ffe4ea99e | up | enabled |
| ac1761a6-6ffe-4527-9a2f-1042a101cf57 | ac1761a6-6ffe-4527-9a2f-1042a101cf57 | up | enabled |
| 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | 3234a4c9-1d0c-4d9c-aa2f-9e4c1390741f | up | enabled |
| 19bbf021-76b4-4222-a633-4539a2c70225 | 19bbf021-76b4-4222-a633-4539a2c70225 | up | enabled |
| 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | 3e37dd34-d6d4-4726-93cb-ad4e293835f8 | up | enabled |
| bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | bb1f5894-7d4c-467a-9b5c-57b6e895ba3b | up | enabled |
| 9cf4ee11-6caa-4c76-9322-1312809b378e | 9cf4ee11-6caa-4c76-9322-1312809b378e | up | enabled |
| fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | fb729bc5-8d29-47b8-8d0f-cbeb47ba57a8 | up | enabled |
| 9d88099a-fb45-4f0b-aded-d1b87236d76f | 9d88099a-fb45-4f0b-aded-d1b87236d76f | up | enabled |
| fdee4953-5e75-47dd-8cbb-755fd746bd46 | fdee4953-5e75-47dd-8cbb-755fd746bd46 | up | enabled |
| 78676b96-a4fb-4dd6-9aaa-1930732aeeda | 78676b96-a4fb-4dd6-9aaa-1930732aeeda | up | enabled |
| 817a882b-8690-43a6-b310-3fb42a4b07ba | 817a882b-8690-43a6-b310-3fb42a4b07ba | up | enabled |
| 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | 60c49aa7-ec65-42d4-b82e-5dc2b3e6c56f | up | enabled |
| e693c0fc-f714-453b-b027-bd62a08c3f4c | e693c0fc-f714-453b-b027-bd62a08c3f4c | up | enabled |
| 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | 5b0118c4-0aa4-4c96-9919-e6beb4e886f2 | up | enabled |
| 89abe9f3-59d3-4e8e-a460-1814cc759b89 | 89abe9f3-59d3-4e8e-a460-1814cc759b89 | up | enabled |
+--------------------------------------+--------------------------------------+-------+----------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056756/+subscriptions
Follow ups
