← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1834527] [NEW] disk_available_least value updates when instance moved but not to the value expected

 

Public bug reported:

Brief Description
-----------------
Originating host updates disk_available_least value whenthe instance migrates off of it (in a resize operation) 
But the value is not what was expected.

nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]

Severity
--------
standard

Steps to Reproduce
------------------

1. Create a flavor flavor-3 with disk size 7
(7f392fd5-bb05-4c4d-bf03-7da019c5bb64)
--vcpus=1 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-3  hw:mem_page_size': '2048
Boot instance using flavor-3
Instance a61b7899-10ad-4121-87e5-d56674f4dfa3 is created and lands on compute-2

see nova compute.log

{"log":"2019-06-27 17:33:59.078 2813101 INFO nova.compute.claims [req-
3da9ec9b-56aa-426a-8f55-312dfa25c85a ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node
compute-2: memory 1024 MB, disk 7 GB, vcpus 1
CPU\n","stream":"stdout","time":"2019-06-27T17:33:59.079843317Z"}


3.  Create another flavor-4 to occupy vcpus that remain on the same host  (flavor-4 53ae6df0-654d-4e3c-82f4-d08ef2ac9a40)
--vcpus=21 --ram=1024 --disk=2 flavor-4 hw:mem_page_size': '2048'

Boot the instance with this flavor to occupy the remaining vcpus on the
same host (compute-2 in this case)

see nova-compute.log

{"log":"2019-06-27 17:34:47.111 2813101 INFO nova.compute.claims [req-
c0b7716e-11bc-4c7e-bf28-d0a512302524 ccc4888ba22547f4a8811237d1c038fc
5a6966f44a474d64ace047ff1d196a20 - default default] [instance: a8a1b43e-
127f-4e47-9476-64cdeaf2ca33] Attempting claim on node compute-2: memory
1024 MB, disk 2 GB, vcpus 21
CPU\n","stream":"stdout","time":"2019-06-27T17:34:47.112205534Z"}


@ [2019-06-27 17:34:44,124]
--poll --flavor=53ae6df0-654d-4e3c-82f4-d08ef2ac9a40 --key-name=keypair-tenant1 --availability-zone=nova:compute-2 --image=bc3f6915-22c4-44c7-a28b-d5fdea648198 --nic net-id=30010e24-2c88-483a-a137-3d4d2c6909e2 --nic net-id=01cfa21b-c5bd-4e35-aaf0-27665de58d2a tenant1-vm-2


5. Check disk_available_least on compute-2 prior to the resize of the first instance that was created in step 1
openstack hypervisor show on compute-2

disk_available_least value is 339 at this point @ [2019-06-27
17:35:17,282]

6. Create another flavor (a resize target flavor)
flavor-5 (6f682e05-ac28-4450-8b89-3671d0166b19)
--vcpus=2 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-5

Resize the instance to flavor-5 and confirm the instance is moved to a different host (resized off of compute-2) 
Resizing VM a61b7899-10ad-4121-87e5-d56674f4dfa3 to flavor 6f682e05-ac28-4450-8b89-3671d0166b19

The instance instance-00000011 is now running on compute-3 with flavor-5 (with disk size 7)
                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-3                                                  |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000011                   

7. Run openstack hypervisor show on compute-2 to confirm the disk_available_least value is returning down to expected value.
$ openstack hypervisor show compute-2
disk_available_least value now appears to be 344

| disk_available_least | 344


Expected Behavior
------------------
Expected disk_available_leasst to update to ~346 (on the originating host compute-2)

339 (compute-2 disk_available_least value before resize operation) 
+ 7 (disk size of instance that moved away) 
= 346 (give or take 1)


Actual Behavior
----------------
---This testcase failed as disk_available_least on compute-2 is now 344
ie. the actual disk_available actually updated on the VM originating host to 344 ie. change of difference of 5 (as opposed to 7)

339 (compute-2 disk_available_least value before resize operation) 
+ 5 (disk size of instance that moved away) 
=344


see nfv-vim.log

2019-06-27T17:35:59.834 controller-0                           VIM_Thread[3129659]     INFO          _instance_state_resize_confirm.py.35    Exiting state (resize-confirm) for tenant1-vm-1.
...

2019-06-27T17:36:00.134 controller-0                           VIM_Thread[3129659]    DEBUG                        _vim_nfvi_events.py.288   Instance action-change, uuid=a61b7899-10ad-4121-87e5-d56674f4dfa3, nfvi_action=resize, nfvi_action_state=started, reason=None.
...
2019-06-27T17:36:02.647 controller-0                           VIM_Thread[3129659]     INFO                      _instance_director.py.1601  Instance tenant1-vm-1 has recovered on host compute-3.
  


compute.log on compute-3

"log":"2019-06-27 17:35:34.056 3011232 INFO nova.compute.claims [req-92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68 bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance: a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node compute-3: memory 1024 MB, disk 7 GB, vcpus 2 CPU\n","stream":"stdout","time":"2019-06-27T17:35:34.057401328Z"}
...

{"log":"2019-06-27 17:35:34.064 3011232 INFO nova.compute.claims [req-
92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68
bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance:
a61b7899-10ad-4121-87e5-d56674f4dfa3] Claim successful on node
compute-3\n","stream":"stdout","time":"2019-06-27T17:35:34.064449551Z"}


Reproducibility
---------------
yes

System Configuration
--------------------
storage
(Lab: PV-0 nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]

Branch/Pull Time/Commit
--------------------
Load: 20190620T013000Z
Job: STX_build_master_master

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1834527

Title:
  disk_available_least value updates when instance moved but not to the
  value expected

Status in OpenStack Compute (nova):
  New

Bug description:
  Brief Description
  -----------------
  Originating host updates disk_available_least value whenthe instance migrates off of it (in a resize operation) 
  But the value is not what was expected.

  nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]

  Severity
  --------
  standard

  Steps to Reproduce
  ------------------

  1. Create a flavor flavor-3 with disk size 7
  (7f392fd5-bb05-4c4d-bf03-7da019c5bb64)
  --vcpus=1 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-3  hw:mem_page_size': '2048
  Boot instance using flavor-3
  Instance a61b7899-10ad-4121-87e5-d56674f4dfa3 is created and lands on compute-2

  see nova compute.log

  {"log":"2019-06-27 17:33:59.078 2813101 INFO nova.compute.claims [req-
  3da9ec9b-56aa-426a-8f55-312dfa25c85a ccc4888ba22547f4a8811237d1c038fc
  5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
  a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node
  compute-2: memory 1024 MB, disk 7 GB, vcpus 1
  CPU\n","stream":"stdout","time":"2019-06-27T17:33:59.079843317Z"}

  
  3.  Create another flavor-4 to occupy vcpus that remain on the same host  (flavor-4 53ae6df0-654d-4e3c-82f4-d08ef2ac9a40)
  --vcpus=21 --ram=1024 --disk=2 flavor-4 hw:mem_page_size': '2048'

  Boot the instance with this flavor to occupy the remaining vcpus on
  the same host (compute-2 in this case)

  see nova-compute.log

  {"log":"2019-06-27 17:34:47.111 2813101 INFO nova.compute.claims [req-
  c0b7716e-11bc-4c7e-bf28-d0a512302524 ccc4888ba22547f4a8811237d1c038fc
  5a6966f44a474d64ace047ff1d196a20 - default default] [instance:
  a8a1b43e-127f-4e47-9476-64cdeaf2ca33] Attempting claim on node
  compute-2: memory 1024 MB, disk 2 GB, vcpus 21
  CPU\n","stream":"stdout","time":"2019-06-27T17:34:47.112205534Z"}

  
  @ [2019-06-27 17:34:44,124]
  --poll --flavor=53ae6df0-654d-4e3c-82f4-d08ef2ac9a40 --key-name=keypair-tenant1 --availability-zone=nova:compute-2 --image=bc3f6915-22c4-44c7-a28b-d5fdea648198 --nic net-id=30010e24-2c88-483a-a137-3d4d2c6909e2 --nic net-id=01cfa21b-c5bd-4e35-aaf0-27665de58d2a tenant1-vm-2

  
  5. Check disk_available_least on compute-2 prior to the resize of the first instance that was created in step 1
  openstack hypervisor show on compute-2

  disk_available_least value is 339 at this point @ [2019-06-27
  17:35:17,282]

  6. Create another flavor (a resize target flavor)
  flavor-5 (6f682e05-ac28-4450-8b89-3671d0166b19)
  --vcpus=2 --ephemeral=0 --swap=0 --ram=1024 --disk=7 flavor-5

  Resize the instance to flavor-5 and confirm the instance is moved to a different host (resized off of compute-2) 
  Resizing VM a61b7899-10ad-4121-87e5-d56674f4dfa3 to flavor 6f682e05-ac28-4450-8b89-3671d0166b19

  The instance instance-00000011 is now running on compute-3 with flavor-5 (with disk size 7)
                                               |
  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-3                                                  |
  | OS-EXT-SRV-ATTR:instance_name       | instance-00000011                   

  7. Run openstack hypervisor show on compute-2 to confirm the disk_available_least value is returning down to expected value.
  $ openstack hypervisor show compute-2
  disk_available_least value now appears to be 344

  | disk_available_least | 344


  Expected Behavior
  ------------------
  Expected disk_available_leasst to update to ~346 (on the originating host compute-2)

  339 (compute-2 disk_available_least value before resize operation) 
  + 7 (disk size of instance that moved away) 
  = 346 (give or take 1)

  
  Actual Behavior
  ----------------
  ---This testcase failed as disk_available_least on compute-2 is now 344
  ie. the actual disk_available actually updated on the VM originating host to 344 ie. change of difference of 5 (as opposed to 7)

  339 (compute-2 disk_available_least value before resize operation) 
  + 5 (disk size of instance that moved away) 
  =344


  see nfv-vim.log

  2019-06-27T17:35:59.834 controller-0                           VIM_Thread[3129659]     INFO          _instance_state_resize_confirm.py.35    Exiting state (resize-confirm) for tenant1-vm-1.
  ...

  2019-06-27T17:36:00.134 controller-0                           VIM_Thread[3129659]    DEBUG                        _vim_nfvi_events.py.288   Instance action-change, uuid=a61b7899-10ad-4121-87e5-d56674f4dfa3, nfvi_action=resize, nfvi_action_state=started, reason=None.
  ...
  2019-06-27T17:36:02.647 controller-0                           VIM_Thread[3129659]     INFO                      _instance_director.py.1601  Instance tenant1-vm-1 has recovered on host compute-3.
    

  
  compute.log on compute-3

  "log":"2019-06-27 17:35:34.056 3011232 INFO nova.compute.claims [req-92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68 bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance: a61b7899-10ad-4121-87e5-d56674f4dfa3] Attempting claim on node compute-3: memory 1024 MB, disk 7 GB, vcpus 2 CPU\n","stream":"stdout","time":"2019-06-27T17:35:34.057401328Z"}
  ...

  {"log":"2019-06-27 17:35:34.064 3011232 INFO nova.compute.claims [req-
  92ca6582-555c-4091-b3ee-fc13189c7dca 85d9be7b82104a9eb54f30de84be7d68
  bef1e6b02a524ae9a6bc50ea21a2b355 - default default] [instance:
  a61b7899-10ad-4121-87e5-d56674f4dfa3] Claim successful on node
  compute-3\n","stream":"stdout","time":"2019-06-27T17:35:34.064449551Z"}


  
  Reproducibility
  ---------------
  yes

  System Configuration
  --------------------
  storage
  (Lab: PV-0 nova/test_resize_vm.py::TestResizeDiffHost::test_resize_different_comp_node[local_image]

  Branch/Pull Time/Commit
  --------------------
  Load: 20190620T013000Z
  Job: STX_build_master_master

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1834527/+subscriptions


Follow ups