yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #81053
[Bug 1856902] [NEW] nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmem fails intermittently
Public bug reported:
Seen here:
https://748441c9eeeb89eb36db-
b17eeb704755458252956ad3afd603fe.ssl.cf5.rackcdn.com/697670/3/gate/nova-
tox-functional-py36/54fe49d/testr_results.html.gz
ft1.1: nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmemtesttools.testresult.real._StringException: pythonlogging:'': {{{
2019-12-18 17:05:18,534 WARNING [placement.db_api] TransactionFactory already started, not reconfiguring.
2019-12-18 17:05:18,885 INFO [nova.service] Starting conductor node (version 20.1.0)
2019-12-18 17:05:18,917 INFO [nova.service] Starting scheduler node (version 20.1.0)
2019-12-18 17:05:18,981 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/flavors/detail" status: 200 len: 2605 microversion: 2.81 time: 0.020144
2019-12-18 17:05:18,996 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/flavors" status: 200 len: 464 microversion: 2.81 time: 0.012445
2019-12-18 17:05:19,018 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/flavors/99512851/os-extra_specs" status: 200 len: 37 microversion: 2.81 time: 0.018592
2019-12-18 17:05:19,035 INFO [nova.virt.driver] Loading compute driver 'libvirt.LibvirtDriver'
2019-12-18 17:05:19,038 WARNING [os_brick.initiator.connectors.remotefs] Connection details not present. RemoteFsClient may not initialize properly.
2019-12-18 17:05:19,040 INFO [nova.service] Starting compute node (version 20.1.0)
2019-12-18 17:05:19,058 WARNING [nova.compute.manager] Compute node host1 not found in the database. If this is the first time this service is starting on this host, then you can ignore this warning.
2019-12-18 17:05:19,063 INFO [nova.compute.manager] Looking for unclaimed instances stuck in BUILDING status for nodes managed by this host
2019-12-18 17:05:19,073 WARNING [nova.compute.manager] No compute node record found for host host1. If this is the first time this service is starting on this host, then you can ignore this warning.
2019-12-18 17:05:19,082 WARNING [nova.compute.resource_tracker] No compute node record for host1:host1
2019-12-18 17:05:19,086 INFO [nova.compute.resource_tracker] Compute node record created for host1:host1 with uuid: 56a25de3-2d91-48a3-9e77-a66f1b5d65b5
2019-12-18 17:05:19,123 INFO [placement.requestlog] 127.0.0.1 "GET /placement/resource_providers?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5" status: 200 len: 26 microversion: 1.14
2019-12-18 17:05:19,131 INFO [placement.requestlog] 127.0.0.1 "POST /placement/resource_providers" status: 200 len: 828 microversion: 1.20
2019-12-18 17:05:19,132 INFO [nova.scheduler.client.report] [req-54d75b29-ade3-4f2a-8911-d1147ef449fc] Created resource provider record via placement API for resource provider with UUID 56a25de3-2d91-48a3-9e77-a66f1b5d65b5 and name host1.
2019-12-18 17:05:19,133 INFO [nova.virt.libvirt.host] kernel doesn't support AMD SEV
2019-12-18 17:05:19,149 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_classes/CUSTOM_PMEM_NAMESPACE_SMALL" status: 201 len: 0 microversion: 1.7
2019-12-18 17:05:19,159 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_classes/CUSTOM_PMEM_NAMESPACE_4GB" status: 201 len: 0 microversion: 1.7
2019-12-18 17:05:19,173 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories" status: 200 len: 665 microversion: 1.26
2019-12-18 17:05:19,181 INFO [placement.requestlog] 127.0.0.1 "GET /placement/traits?name=in:COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_AMI,HW_CPU_X86_VMX,HW_CPU_X86_AESNI,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_RAW,COMPUTE_DEVICE_TAGGING,HW_CPU_HYPERTHREADING,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_TRUSTED_CERTS,COMPUTE_NODE" status: 200 len: 432 microversion: 1.6
2019-12-18 17:05:19,200 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/traits" status: 200 len: 467 microversion: 1.6
2019-12-18 17:05:19,224 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133 microversion: 2.81 time: 0.019077
2019-12-18 17:05:19,236 INFO [placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories" status: 200 len: 665 microversion: 1.0
2019-12-18 17:05:19,253 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133 microversion: 2.81 time: 0.014420
2019-12-18 17:05:19,255 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:19,329 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.071830
2019-12-18 17:05:19,352 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1778 microversion: 2.81 time: 0.019663
2019-12-18 17:05:19,473 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 1087 microversion: 1.34
2019-12-18 17:05:19,495 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to host1
2019-12-18 17:05:19,503 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 19 microversion: 1.28
2019-12-18 17:05:19,530 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 204 len: 0 microversion: 1.34
2019-12-18 17:05:19,724 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:19,731 INFO [nova.compute.claims] Claim successful on node host1
2019-12-18 17:05:19,985 INFO [nova.compute.manager] Took 0.00 seconds to spawn the instance on the hypervisor.
2019-12-18 17:05:20,077 INFO [nova.compute.manager] Took 0.36 seconds to build instance.
2019-12-18 17:05:20,087 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1875 microversion: 2.81 time: 0.227589
2019-12-18 17:05:20,122 INFO [nova.scheduler.host_manager] Received an update from an unknown host 'host1'. Re-created its InstanceList.
2019-12-18 17:05:20,679 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1880 microversion: 2.81 time: 0.087029
2019-12-18 17:05:20,690 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 169 microversion: 1.0
2019-12-18 17:05:20,691 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:20,766 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.072347
2019-12-18 17:05:20,786 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1778 microversion: 2.81 time: 0.017267
2019-12-18 17:05:20,902 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 1090 microversion: 1.34
2019-12-18 17:05:20,922 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to host1
2019-12-18 17:05:20,930 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 19 microversion: 1.28
2019-12-18 17:05:20,955 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 1.34
2019-12-18 17:05:21,131 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:21,138 INFO [nova.compute.claims] Claim successful on node host1
2019-12-18 17:05:21,391 INFO [nova.compute.manager] Took 0.00 seconds to spawn the instance on the hypervisor.
2019-12-18 17:05:21,450 INFO [nova.compute.manager] Took 0.33 seconds to build instance.
2019-12-18 17:05:21,490 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1863 microversion: 2.81 time: 0.183892
2019-12-18 17:05:22,075 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1880 microversion: 2.81 time: 0.080401
2019-12-18 17:05:22,084 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 169 microversion: 1.0
2019-12-18 17:05:22,085 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:22,162 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.074114
2019-12-18 17:05:22,183 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/afeb21d7-a89d-40ea-ac6e-28a7b80cec95" status: 200 len: 1778 microversion: 2.81 time: 0.018034
2019-12-18 17:05:22,283 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 53 microversion: 1.34
2019-12-18 17:05:22,284 INFO [nova.scheduler.manager] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
2019-12-18 17:05:22,284 ERROR [nova.conductor.manager] Failed to schedule instances
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances
instance_uuids, return_alternates=True)
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances
return_alternates=return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations
instance_uuids, return_objects, return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations
return cctxt.call(ctxt, 'select_destinations', **msg_args)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
transport_options=self.transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
transport_options=transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send
raise failure
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner
return func(*args, **kwargs)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations
raise exception.NoValidHost(reason="")
nova.exception.NoValidHost: No valid host was found.
2019-12-18 17:05:22,325 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: No valid host was found.
2019-12-18 17:05:22,325 WARNING [nova.scheduler.utils] Setting instance to ERROR state.
2019-12-18 17:05:22,773 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/afeb21d7-a89d-40ea-ac6e-28a7b80cec95" status: 200 len: 3749 microversion: 2.81 time: 0.084113
2019-12-18 17:05:22,899 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 2.81 time: 0.122724
2019-12-18 17:05:22,901 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:22,952 INFO [nova.compute.manager] Terminating instance
2019-12-18 17:05:22,968 INFO [nova.virt.libvirt.driver] Instance destroyed successfully.
2019-12-18 17:05:22,976 INFO [nova.virt.libvirt.driver] Deletion of /home/zuul/src/opendev.org/openstack/nova/instances/68f4cbad-7c69-492a-a173-6869dfa312d7_del complete
2019-12-18 17:05:23,042 INFO [nova.compute.manager] Took 0.08 seconds to destroy the instance on the hypervisor.
2019-12-18 17:05:23,051 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.147582
2019-12-18 17:05:23,077 INFO [nova.compute.manager] Took 0.02 seconds to deallocate network for instance.
2019-12-18 17:05:23,095 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 1778 microversion: 2.81 time: 0.040906
2019-12-18 17:05:23,179 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:23,204 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 53 microversion: 1.34
2019-12-18 17:05:23,205 INFO [nova.scheduler.manager] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
2019-12-18 17:05:23,208 ERROR [nova.conductor.manager] Failed to schedule instances
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances
instance_uuids, return_alternates=True)
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances
return_alternates=return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations
instance_uuids, return_objects, return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations
return cctxt.call(ctxt, 'select_destinations', **msg_args)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
transport_options=self.transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
transport_options=transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send
raise failure
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner
return func(*args, **kwargs)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations
raise exception.NoValidHost(reason="")
nova.exception.NoValidHost: No valid host was found.
2019-12-18 17:05:23,240 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 1.28
2019-12-18 17:05:23,241 INFO [nova.scheduler.client.report] Deleted allocation for instance 68f4cbad-7c69-492a-a173-6869dfa312d7
2019-12-18 17:05:23,312 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: No valid host was found.
2019-12-18 17:05:23,313 WARNING [nova.scheduler.utils] Setting instance to ERROR state.
2019-12-18 17:05:23,696 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.094132
2019-12-18 17:05:24,254 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.053813
2019-12-18 17:05:24,817 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.058160
2019-12-18 17:05:25,376 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.054598
2019-12-18 17:05:25,943 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.062802
2019-12-18 17:05:26,502 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.054831
2019-12-18 17:05:27,067 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.060461
2019-12-18 17:05:27,629 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.057168
2019-12-18 17:05:28,199 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.065318
2019-12-18 17:05:28,339 INFO [oslo_db.sqlalchemy.provision] DROP ALL OBJECTS, BACKEND sqlite://
2019-12-18 17:05:28,372 INFO [oslo_db.sqlalchemy.provision] DISPOSE ENGINE Engine(sqlite://)
}}}
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/libvirt/test_vpmem.py", line 191, in test_create_servers_with_vpmem
self._wait_for_state_change(server3, 'ACTIVE')
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/integrated_helpers.py", line 98, in _wait_for_state_change
server, {'status': expected_status}, max_retries)
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/integrated_helpers.py", line 91, in _wait_for_server_parameter
expected_params, server))
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/unittest2/case.py", line 690, in fail
raise self.failureException(msg)
AssertionError: Wait for state change failed, expected_params={'status': 'ACTIVE'}, server={'id': 'e67add72-1327-466a-8f96-7e5025fb9847', 'name': 'some-server', 'status': 'ERROR', 'tenant_id': '6f70656e737461636b20342065766572', 'user_id': 'admin', 'metadata': {}, 'hostId': '', 'image': {'id': '155d900f-4e14-4e4c-a73d-069cbf4541e6', 'links': [{'rel': 'bookmark', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/images/155d900f-4e14-4e4c-a73d-069cbf4541e6'}]}, 'flavor': {'vcpus': 1, 'ram': 1024, 'disk': 1, 'ephemeral': 0, 'swap': 0, 'original_name': 'flavorBGI1I584', 'extra_specs': {'hw:pmem': 'SMALL'}}, 'created': '2019-12-18T17:05:23Z', 'updated': '2019-12-18T17:05:23Z', 'addresses': {}, 'accessIPv4': '', 'accessIPv6': '', 'links': [{'rel': 'self', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847'}, {'rel': 'bookmark', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/servers/e67add72-1327-466a-8f96-7e5025fb9847'}], 'OS-DCF:diskConfig': 'MANUAL', 'fault': {'code': 500, 'created': '2019-12-18T17:05:23Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances\n return_alternates=return_alternates)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send\n raise failure\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner\n return func(*args, **kwargs)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations\n raise exception.NoValidHost(reason="")\nnova.exception.NoValidHost: No valid host was found. \n'}, 'OS-EXT-AZ:availability_zone': 'nova', 'config_drive': '', 'key_name': None, 'OS-SRV-USG:launched_at': None, 'OS-SRV-USG:terminated_at': None, 'OS-EXT-SRV-ATTR:host': None, 'OS-EXT-SRV-ATTR:instance_name': 'instance-00000002', 'OS-EXT-SRV-ATTR:hypervisor_hostname': None, 'OS-EXT-SRV-ATTR:reservation_id': 'r-e4t4h5f5', 'OS-EXT-SRV-ATTR:launch_index': 0, 'OS-EXT-SRV-ATTR:hostname': 'some-server', 'OS-EXT-SRV-ATTR:kernel_id': '', 'OS-EXT-SRV-ATTR:ramdisk_id': '', 'OS-EXT-SRV-ATTR:root_device_name': None, 'OS-EXT-SRV-ATTR:user_data': None, 'OS-EXT-STS:task_state': None, 'OS-EXT-STS:vm_state': 'error', 'OS-EXT-STS:power_state': 0, 'os-extended-volumes:volumes_attached': [], 'host_status': '', 'locked': False, 'locked_reason': None, 'description': None, 'tags': [], 'trusted_image_certificates': None, 'server_groups': []}
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22'extra_specs'%3A%20%7B'hw%3Apmem'%3A%20'SMALL'%7D%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
18 hits in 7 days, check and gate, all failures.
** Affects: nova
Importance: Medium
Status: Confirmed
** Tags: gate-failure testing
** Changed in: nova
Status: New => Confirmed
** Changed in: nova
Importance: Undecided => Medium
** Tags added: gate-failure testing
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1856902
Title:
nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmem
fails intermittently
Status in OpenStack Compute (nova):
Confirmed
Bug description:
Seen here:
https://748441c9eeeb89eb36db-
b17eeb704755458252956ad3afd603fe.ssl.cf5.rackcdn.com/697670/3/gate
/nova-tox-functional-py36/54fe49d/testr_results.html.gz
ft1.1: nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmemtesttools.testresult.real._StringException: pythonlogging:'': {{{
2019-12-18 17:05:18,534 WARNING [placement.db_api] TransactionFactory already started, not reconfiguring.
2019-12-18 17:05:18,885 INFO [nova.service] Starting conductor node (version 20.1.0)
2019-12-18 17:05:18,917 INFO [nova.service] Starting scheduler node (version 20.1.0)
2019-12-18 17:05:18,981 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/flavors/detail" status: 200 len: 2605 microversion: 2.81 time: 0.020144
2019-12-18 17:05:18,996 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/flavors" status: 200 len: 464 microversion: 2.81 time: 0.012445
2019-12-18 17:05:19,018 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/flavors/99512851/os-extra_specs" status: 200 len: 37 microversion: 2.81 time: 0.018592
2019-12-18 17:05:19,035 INFO [nova.virt.driver] Loading compute driver 'libvirt.LibvirtDriver'
2019-12-18 17:05:19,038 WARNING [os_brick.initiator.connectors.remotefs] Connection details not present. RemoteFsClient may not initialize properly.
2019-12-18 17:05:19,040 INFO [nova.service] Starting compute node (version 20.1.0)
2019-12-18 17:05:19,058 WARNING [nova.compute.manager] Compute node host1 not found in the database. If this is the first time this service is starting on this host, then you can ignore this warning.
2019-12-18 17:05:19,063 INFO [nova.compute.manager] Looking for unclaimed instances stuck in BUILDING status for nodes managed by this host
2019-12-18 17:05:19,073 WARNING [nova.compute.manager] No compute node record found for host host1. If this is the first time this service is starting on this host, then you can ignore this warning.
2019-12-18 17:05:19,082 WARNING [nova.compute.resource_tracker] No compute node record for host1:host1
2019-12-18 17:05:19,086 INFO [nova.compute.resource_tracker] Compute node record created for host1:host1 with uuid: 56a25de3-2d91-48a3-9e77-a66f1b5d65b5
2019-12-18 17:05:19,123 INFO [placement.requestlog] 127.0.0.1 "GET /placement/resource_providers?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5" status: 200 len: 26 microversion: 1.14
2019-12-18 17:05:19,131 INFO [placement.requestlog] 127.0.0.1 "POST /placement/resource_providers" status: 200 len: 828 microversion: 1.20
2019-12-18 17:05:19,132 INFO [nova.scheduler.client.report] [req-54d75b29-ade3-4f2a-8911-d1147ef449fc] Created resource provider record via placement API for resource provider with UUID 56a25de3-2d91-48a3-9e77-a66f1b5d65b5 and name host1.
2019-12-18 17:05:19,133 INFO [nova.virt.libvirt.host] kernel doesn't support AMD SEV
2019-12-18 17:05:19,149 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_classes/CUSTOM_PMEM_NAMESPACE_SMALL" status: 201 len: 0 microversion: 1.7
2019-12-18 17:05:19,159 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_classes/CUSTOM_PMEM_NAMESPACE_4GB" status: 201 len: 0 microversion: 1.7
2019-12-18 17:05:19,173 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories" status: 200 len: 665 microversion: 1.26
2019-12-18 17:05:19,181 INFO [placement.requestlog] 127.0.0.1 "GET /placement/traits?name=in:COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_AMI,HW_CPU_X86_VMX,HW_CPU_X86_AESNI,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_RAW,COMPUTE_DEVICE_TAGGING,HW_CPU_HYPERTHREADING,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_TRUSTED_CERTS,COMPUTE_NODE" status: 200 len: 432 microversion: 1.6
2019-12-18 17:05:19,200 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/traits" status: 200 len: 467 microversion: 1.6
2019-12-18 17:05:19,224 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133 microversion: 2.81 time: 0.019077
2019-12-18 17:05:19,236 INFO [placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories" status: 200 len: 665 microversion: 1.0
2019-12-18 17:05:19,253 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133 microversion: 2.81 time: 0.014420
2019-12-18 17:05:19,255 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:19,329 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.071830
2019-12-18 17:05:19,352 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1778 microversion: 2.81 time: 0.019663
2019-12-18 17:05:19,473 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 1087 microversion: 1.34
2019-12-18 17:05:19,495 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to host1
2019-12-18 17:05:19,503 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 19 microversion: 1.28
2019-12-18 17:05:19,530 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 204 len: 0 microversion: 1.34
2019-12-18 17:05:19,724 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:19,731 INFO [nova.compute.claims] Claim successful on node host1
2019-12-18 17:05:19,985 INFO [nova.compute.manager] Took 0.00 seconds to spawn the instance on the hypervisor.
2019-12-18 17:05:20,077 INFO [nova.compute.manager] Took 0.36 seconds to build instance.
2019-12-18 17:05:20,087 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1875 microversion: 2.81 time: 0.227589
2019-12-18 17:05:20,122 INFO [nova.scheduler.host_manager] Received an update from an unknown host 'host1'. Re-created its InstanceList.
2019-12-18 17:05:20,679 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 1880 microversion: 2.81 time: 0.087029
2019-12-18 17:05:20,690 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/3a085e4e-52ba-49cc-be7c-6eca2974765a" status: 200 len: 169 microversion: 1.0
2019-12-18 17:05:20,691 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:20,766 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.072347
2019-12-18 17:05:20,786 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1778 microversion: 2.81 time: 0.017267
2019-12-18 17:05:20,902 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 1090 microversion: 1.34
2019-12-18 17:05:20,922 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to host1
2019-12-18 17:05:20,930 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 19 microversion: 1.28
2019-12-18 17:05:20,955 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 1.34
2019-12-18 17:05:21,131 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:21,138 INFO [nova.compute.claims] Claim successful on node host1
2019-12-18 17:05:21,391 INFO [nova.compute.manager] Took 0.00 seconds to spawn the instance on the hypervisor.
2019-12-18 17:05:21,450 INFO [nova.compute.manager] Took 0.33 seconds to build instance.
2019-12-18 17:05:21,490 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1863 microversion: 2.81 time: 0.183892
2019-12-18 17:05:22,075 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 1880 microversion: 2.81 time: 0.080401
2019-12-18 17:05:22,084 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 169 microversion: 1.0
2019-12-18 17:05:22,085 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:22,162 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.074114
2019-12-18 17:05:22,183 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/afeb21d7-a89d-40ea-ac6e-28a7b80cec95" status: 200 len: 1778 microversion: 2.81 time: 0.018034
2019-12-18 17:05:22,283 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 53 microversion: 1.34
2019-12-18 17:05:22,284 INFO [nova.scheduler.manager] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
2019-12-18 17:05:22,284 ERROR [nova.conductor.manager] Failed to schedule instances
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances
instance_uuids, return_alternates=True)
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances
return_alternates=return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations
instance_uuids, return_objects, return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations
return cctxt.call(ctxt, 'select_destinations', **msg_args)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
transport_options=self.transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
transport_options=transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send
raise failure
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner
return func(*args, **kwargs)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations
raise exception.NoValidHost(reason="")
nova.exception.NoValidHost: No valid host was found.
2019-12-18 17:05:22,325 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: No valid host was found.
2019-12-18 17:05:22,325 WARNING [nova.scheduler.utils] Setting instance to ERROR state.
2019-12-18 17:05:22,773 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/afeb21d7-a89d-40ea-ac6e-28a7b80cec95" status: 200 len: 3749 microversion: 2.81 time: 0.084113
2019-12-18 17:05:22,899 INFO [nova.api.openstack.requestlog] 127.0.0.1 "DELETE /v2.1/servers/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 2.81 time: 0.122724
2019-12-18 17:05:22,901 INFO [nova.tests.functional.libvirt.test_vpmem] booting on host1
2019-12-18 17:05:22,952 INFO [nova.compute.manager] Terminating instance
2019-12-18 17:05:22,968 INFO [nova.virt.libvirt.driver] Instance destroyed successfully.
2019-12-18 17:05:22,976 INFO [nova.virt.libvirt.driver] Deletion of /home/zuul/src/opendev.org/openstack/nova/instances/68f4cbad-7c69-492a-a173-6869dfa312d7_del complete
2019-12-18 17:05:23,042 INFO [nova.compute.manager] Took 0.08 seconds to destroy the instance on the hypervisor.
2019-12-18 17:05:23,051 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.147582
2019-12-18 17:05:23,077 INFO [nova.compute.manager] Took 0.02 seconds to deallocate network for instance.
2019-12-18 17:05:23,095 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 1778 microversion: 2.81 time: 0.040906
2019-12-18 17:05:23,179 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 200 len: 265 microversion: 1.28
2019-12-18 17:05:23,204 INFO [placement.requestlog] 127.0.0.1 "GET /placement/allocation_candidates?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5&limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=CUSTOM_PMEM_NAMESPACE_SMALL%3A1%2CDISK_GB%3A1%2CMEMORY_MB%3A1024%2CVCPU%3A1" status: 200 len: 53 microversion: 1.34
2019-12-18 17:05:23,205 INFO [nova.scheduler.manager] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
2019-12-18 17:05:23,208 ERROR [nova.conductor.manager] Failed to schedule instances
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances
instance_uuids, return_alternates=True)
File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances
return_alternates=return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations
instance_uuids, return_objects, return_alternates)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations
return cctxt.call(ctxt, 'select_destinations', **msg_args)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
transport_options=self.transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
transport_options=transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
transport_options)
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send
raise failure
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner
return func(*args, **kwargs)
File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations
raise exception.NoValidHost(reason="")
nova.exception.NoValidHost: No valid host was found.
2019-12-18 17:05:23,240 INFO [placement.requestlog] 127.0.0.1 "PUT /placement/allocations/68f4cbad-7c69-492a-a173-6869dfa312d7" status: 204 len: 0 microversion: 1.28
2019-12-18 17:05:23,241 INFO [nova.scheduler.client.report] Deleted allocation for instance 68f4cbad-7c69-492a-a173-6869dfa312d7
2019-12-18 17:05:23,312 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: No valid host was found.
2019-12-18 17:05:23,313 WARNING [nova.scheduler.utils] Setting instance to ERROR state.
2019-12-18 17:05:23,696 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.094132
2019-12-18 17:05:24,254 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.053813
2019-12-18 17:05:24,817 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.058160
2019-12-18 17:05:25,376 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.054598
2019-12-18 17:05:25,943 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.062802
2019-12-18 17:05:26,502 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.054831
2019-12-18 17:05:27,067 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.060461
2019-12-18 17:05:27,629 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.057168
2019-12-18 17:05:28,199 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847" status: 200 len: 3749 microversion: 2.81 time: 0.065318
2019-12-18 17:05:28,339 INFO [oslo_db.sqlalchemy.provision] DROP ALL OBJECTS, BACKEND sqlite://
2019-12-18 17:05:28,372 INFO [oslo_db.sqlalchemy.provision] DISPOSE ENGINE Engine(sqlite://)
}}}
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/libvirt/test_vpmem.py", line 191, in test_create_servers_with_vpmem
self._wait_for_state_change(server3, 'ACTIVE')
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/integrated_helpers.py", line 98, in _wait_for_state_change
server, {'status': expected_status}, max_retries)
File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/integrated_helpers.py", line 91, in _wait_for_server_parameter
expected_params, server))
File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/unittest2/case.py", line 690, in fail
raise self.failureException(msg)
AssertionError: Wait for state change failed, expected_params={'status': 'ACTIVE'}, server={'id': 'e67add72-1327-466a-8f96-7e5025fb9847', 'name': 'some-server', 'status': 'ERROR', 'tenant_id': '6f70656e737461636b20342065766572', 'user_id': 'admin', 'metadata': {}, 'hostId': '', 'image': {'id': '155d900f-4e14-4e4c-a73d-069cbf4541e6', 'links': [{'rel': 'bookmark', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/images/155d900f-4e14-4e4c-a73d-069cbf4541e6'}]}, 'flavor': {'vcpus': 1, 'ram': 1024, 'disk': 1, 'ephemeral': 0, 'swap': 0, 'original_name': 'flavorBGI1I584', 'extra_specs': {'hw:pmem': 'SMALL'}}, 'created': '2019-12-18T17:05:23Z', 'updated': '2019-12-18T17:05:23Z', 'addresses': {}, 'accessIPv4': '', 'accessIPv6': '', 'links': [{'rel': 'self', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/v2.1/servers/e67add72-1327-466a-8f96-7e5025fb9847'}, {'rel': 'bookmark', 'href': 'http://2a542091-42cb-4ba0-a2eb-9ed61b4b29da/servers/e67add72-1327-466a-8f96-7e5025fb9847'}], 'OS-DCF:diskConfig': 'MANUAL', 'fault': {'code': 500, 'created': '2019-12-18T17:05:23Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 1412, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/conductor/manager.py", line 855, in _schedule_instances\n return_alternates=return_alternates)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n transport_options)\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_fake.py", line 208, in _send\n raise failure\n File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner\n return func(*args, **kwargs)\n File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/manager.py", line 199, in select_destinations\n raise exception.NoValidHost(reason="")\nnova.exception.NoValidHost: No valid host was found. \n'}, 'OS-EXT-AZ:availability_zone': 'nova', 'config_drive': '', 'key_name': None, 'OS-SRV-USG:launched_at': None, 'OS-SRV-USG:terminated_at': None, 'OS-EXT-SRV-ATTR:host': None, 'OS-EXT-SRV-ATTR:instance_name': 'instance-00000002', 'OS-EXT-SRV-ATTR:hypervisor_hostname': None, 'OS-EXT-SRV-ATTR:reservation_id': 'r-e4t4h5f5', 'OS-EXT-SRV-ATTR:launch_index': 0, 'OS-EXT-SRV-ATTR:hostname': 'some-server', 'OS-EXT-SRV-ATTR:kernel_id': '', 'OS-EXT-SRV-ATTR:ramdisk_id': '', 'OS-EXT-SRV-ATTR:root_device_name': None, 'OS-EXT-SRV-ATTR:user_data': None, 'OS-EXT-STS:task_state': None, 'OS-EXT-STS:vm_state': 'error', 'OS-EXT-STS:power_state': 0, 'os-extended-volumes:volumes_attached': [], 'host_status': '', 'locked': False, 'locked_reason': None, 'description': None, 'tags': [], 'trusted_image_certificates': None, 'server_groups': []}
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22'extra_specs'%3A%20%7B'hw%3Apmem'%3A%20'SMALL'%7D%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
18 hits in 7 days, check and gate, all failures.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1856902/+subscriptions
Follow ups
