yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #79075
[Bug 1834527] [NEW] disk_available_least value updates when instance moved but not to the value expected
Public bug reported:
Brief Description
-----------------
Originating host updates disk_available_least value whenthe instance migrates off of it (in a resize operation)
But the value is not what was expected.
nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]
Severity
--------
standard
Steps to Reproduce
------------------
1. Create a flavor flavor-3 with disk size 7
(7f392fd5-bb05-4c4d-bf03-7da019c5bb64)
--vcpus=1 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-3 hw:mem_page_size': '2048
Boot instance using flavor-3
Instance a61b7899-10ad-4121-87e5-d56674f4dfa3 is created and lands on compute-2
see nova compute.log
{"log":"2019-06-27 17:33:59.078 2813101 INFO nova.compute.claims [req-
3da9ec9b-56aa-426a-8f55-312dfa25c85a ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node
compute-2: memory 1024 MB, disk 7 GB, vcpus 1
CPU\n","stream":"stdout","time":"2019-06-27T17:33:59.079843317Z"}
3. Create another flavor-4 to occupy vcpus that remain on the same host (flavor-4 53ae6df0-654d-4e3c-82f4-d08ef2ac9a40)
--vcpus=21 --ram=1024 --disk=2 flavor-4 hw:mem_page_size': '2048'
Boot the instance with this flavor to occupy the remaining vcpus on the
same host (compute-2 in this case)
see nova-compute.log
{"log":"2019-06-27 17:34:47.111 2813101 INFO nova.compute.claims [req-
c0b7716e-11bc-4c7e-bf28-d0a512302524 ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance: a8a1b43e-
127f-4e47-9476-64cdeaf2ca33] Attempting claim on node compute-2: memory
1024 MB, disk 2 GB, vcpus 21
CPU\n","stream":"stdout","time":"2019-06-27T17:34:47.112205534Z"}
@ [2019-06-27 17:34:44,124]
--poll --flavor=53ae6df0-654d-4e3c-82f4-d08ef2ac9a40 --key-name=keypair-tenant1 --availability-zone=nova:compute-2 --image=bc3f6915-22c4-44c7-a28b-d5fdea648198 --nic net-id=30010e24-2c88-483a-a137-3d4d2c6909e2 --nic net-id=01cfa21b-c5bd-4e35-aaf0-27665de58d2a tenant1-vm-2
5. Check disk_available_least on compute-2 prior to the resize of the first instance that was created in step 1
openstack hypervisor show on compute-2
disk_available_least value is 339 at this point @ [2019-06-27
17:35:17,282]
6. Create another flavor (a resize target flavor)
flavor-5 (6f682e05-ac28-4450-8b89-3671d0166b19)
--vcpus=2 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-5
Resize the instance to flavor-5 and confirm the instance is moved to a different host (resized off of compute-2)
Resizing VM a61b7899-10ad-4121-87e5-d56674f4dfa3 to flavor 6f682e05-ac28-4450-8b89-3671d0166b19
The instance instance-00000011 is now running on compute-3 with flavor-5 (with disk size 7)
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-3 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000011
7. Run openstack hypervisor show on compute-2 to confirm the disk_available_least value is returning down to expected value.
$ openstack hypervisor show compute-2
disk_available_least value now appears to be 344
| disk_available_least | 344
Expected Behavior
------------------
Expected disk_available_leasst to update to ~346 (on the originating host compute-2)
339 (compute-2 disk_available_least value before resize operation)
+ 7 (disk size of instance that moved away)
= 346 (give or take 1)
Actual Behavior
----------------
---This testcase failed as disk_available_least on compute-2 is now 344
ie. the actual disk_available actually updated on the VM originating host to 344 ie. change of difference of 5 (as opposed to 7)
339 (compute-2 disk_available_least value before resize operation)
+ 5 (disk size of instance that moved away)
=344
see nfv-vim.log
2019-06-27T17:35:59.834 controller-0 VIM_Thread[3129659] INFO _instance_state_resize_confirm.py.35 Exiting state (resize-confirm) for tenant1-vm-1.
...
2019-06-27T17:36:00.134 controller-0 VIM_Thread[3129659] DEBUG _vim_nfvi_events.py.288 Instance action-change, uuid=a61b7899-10ad-4121-87e5-d56674f4dfa3, nfvi_action=resize, nfvi_action_state=started, reason=None.
...
2019-06-27T17:36:02.647 controller-0 VIM_Thread[3129659] INFO _instance_director.py.1601 Instance tenant1-vm-1 has recovered on host compute-3.
compute.log on compute-3
"log":"2019-06-27 17:35:34.056 3011232 INFO nova.compute.claims [req-92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68 bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance: a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node compute-3: memory 1024 MB, disk 7 GB, vcpus 2 CPU\n","stream":"stdout","time":"2019-06-27T17:35:34.057401328Z"}
...
{"log":"2019-06-27 17:35:34.064 3011232 INFO nova.compute.claims [req-
92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68
bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Claim successful on node
compute-3\n","stream":"stdout","time":"2019-06-27T17:35:34.064449551Z"}
Reproducibility
---------------
yes
System Configuration
--------------------
storage
(Lab: PV-0 nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]
Branch/Pull Time/Commit
--------------------
Load: 20190620T013000Z
Job: STX_build_master_master
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1834527
Title:
disk_available_least value updates when instance moved but not to the
value expected
Status in OpenStack Compute (nova):
New
Bug description:
Brief Description
-----------------
Originating host updates disk_available_least value whenthe instance migrates off of it (in a resize operation)
But the value is not what was expected.
nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]
Severity
--------
standard
Steps to Reproduce
------------------
1. Create a flavor flavor-3 with disk size 7
(7f392fd5-bb05-4c4d-bf03-7da019c5bb64)
--vcpus=1 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-3 hw:mem_page_size': '2048
Boot instance using flavor-3
Instance a61b7899-10ad-4121-87e5-d56674f4dfa3 is created and lands on compute-2
see nova compute.log
{"log":"2019-06-27 17:33:59.078 2813101 INFO nova.compute.claims [req-
3da9ec9b-56aa-426a-8f55-312dfa25c85a ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node
compute-2: memory 1024 MB, disk 7 GB, vcpus 1
CPU\n","stream":"stdout","time":"2019-06-27T17:33:59.079843317Z"}
3. Create another flavor-4 to occupy vcpus that remain on the same host (flavor-4 53ae6df0-654d-4e3c-82f4-d08ef2ac9a40)
--vcpus=21 --ram=1024 --disk=2 flavor-4 hw:mem_page_size': '2048'
Boot the instance with this flavor to occupy the remaining vcpus on
the same host (compute-2 in this case)
see nova-compute.log
{"log":"2019-06-27 17:34:47.111 2813101 INFO nova.compute.claims [req-
c0b7716e-11bc-4c7e-bf28-d0a512302524 ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
a8a1b43e-127f-4e47-9476-64cdeaf2ca33] Attempting claim on node
compute-2: memory 1024 MB, disk 2 GB, vcpus 21
CPU\n","stream":"stdout","time":"2019-06-27T17:34:47.112205534Z"}
@ [2019-06-27 17:34:44,124]
--poll --flavor=53ae6df0-654d-4e3c-82f4-d08ef2ac9a40 --key-name=keypair-tenant1 --availability-zone=nova:compute-2 --image=bc3f6915-22c4-44c7-a28b-d5fdea648198 --nic net-id=30010e24-2c88-483a-a137-3d4d2c6909e2 --nic net-id=01cfa21b-c5bd-4e35-aaf0-27665de58d2a tenant1-vm-2
5. Check disk_available_least on compute-2 prior to the resize of the first instance that was created in step 1
openstack hypervisor show on compute-2
disk_available_least value is 339 at this point @ [2019-06-27
17:35:17,282]
6. Create another flavor (a resize target flavor)
flavor-5 (6f682e05-ac28-4450-8b89-3671d0166b19)
--vcpus=2 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-5
Resize the instance to flavor-5 and confirm the instance is moved to a different host (resized off of compute-2)
Resizing VM a61b7899-10ad-4121-87e5-d56674f4dfa3 to flavor 6f682e05-ac28-4450-8b89-3671d0166b19
The instance instance-00000011 is now running on compute-3 with flavor-5 (with disk size 7)
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-3 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000011
7. Run openstack hypervisor show on compute-2 to confirm the disk_available_least value is returning down to expected value.
$ openstack hypervisor show compute-2
disk_available_least value now appears to be 344
| disk_available_least | 344
Expected Behavior
------------------
Expected disk_available_leasst to update to ~346 (on the originating host compute-2)
339 (compute-2 disk_available_least value before resize operation)
+ 7 (disk size of instance that moved away)
= 346 (give or take 1)
Actual Behavior
----------------
---This testcase failed as disk_available_least on compute-2 is now 344
ie. the actual disk_available actually updated on the VM originating host to 344 ie. change of difference of 5 (as opposed to 7)
339 (compute-2 disk_available_least value before resize operation)
+ 5 (disk size of instance that moved away)
=344
see nfv-vim.log
2019-06-27T17:35:59.834 controller-0 VIM_Thread[3129659] INFO _instance_state_resize_confirm.py.35 Exiting state (resize-confirm) for tenant1-vm-1.
...
2019-06-27T17:36:00.134 controller-0 VIM_Thread[3129659] DEBUG _vim_nfvi_events.py.288 Instance action-change, uuid=a61b7899-10ad-4121-87e5-d56674f4dfa3, nfvi_action=resize, nfvi_action_state=started, reason=None.
...
2019-06-27T17:36:02.647 controller-0 VIM_Thread[3129659] INFO _instance_director.py.1601 Instance tenant1-vm-1 has recovered on host compute-3.
compute.log on compute-3
"log":"2019-06-27 17:35:34.056 3011232 INFO nova.compute.claims [req-92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68 bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance: a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node compute-3: memory 1024 MB, disk 7 GB, vcpus 2 CPU\n","stream":"stdout","time":"2019-06-27T17:35:34.057401328Z"}
...
{"log":"2019-06-27 17:35:34.064 3011232 INFO nova.compute.claims [req-
92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68
bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Claim successful on node
compute-3\n","stream":"stdout","time":"2019-06-27T17:35:34.064449551Z"}
Reproducibility
---------------
yes
System Configuration
--------------------
storage
(Lab: PV-0 nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]
Branch/Pull Time/Commit
--------------------
Load: 20190620T013000Z
Job: STX_build_master_master
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1834527/+subscriptions
Follow ups
