yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #93846
[Bug 2061569] [NEW] Live migration fails despite identical CPUs with Host filter ignoring hosts
Public bug reported:
charm bundle: openstack-2023.2
nova-compute: 28.0.1 channel 2023.2/stable
ubuntu: jammy .15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
I have two identical host, op1 and op4, DELL PowerEdge R7425 acting as nova-compute nodes (hypervisors).
ubuntu@juju:~/juju/controller$ openstack hypervisor list
+----+---------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+--------------+-------+
| 7 | op1.maas | QEMU | xxxxxxxxxxxx | up |
| 8 | op4.maas | QEMU | xxxxxxxxxxxx | up |
+----+---------------------+-----------------+--------------+-------+
1)microcode is the same:
root@op1:~# virsh capabilities |grep microcode
<microcode version='134222446'/>
root@op4:~# virsh capabilities |grep microcode
<microcode version='134222446'/>
2) libvirt capabilities features are identical:
root@op1:~# virsh capabilities |grep feature
<feature name='ht'/>
<feature name='monitor'/>
<feature name='osxsave'/>
<feature name='xsaves'/>
<feature name='cmp_legacy'/>
<feature name='extapic'/>
<feature name='skinit'/>
<feature name='wdt'/>
<feature name='tce'/>
<feature name='topoext'/>
<feature name='perfctr_core'/>
<feature name='perfctr_nb'/>
<feature name='invtsc'/>
<feature name='clzero'/>
<feature name='xsaveerptr'/>
<feature name='npt'/>
<feature name='lbrv'/>
<feature name='svm-lock'/>
<feature name='nrip-save'/>
<feature name='tsc-scale'/>
<feature name='vmcb-clean'/>
<feature name='flushbyasid'/>
<feature name='decodeassists'/>
<feature name='pause-filter'/>
<feature name='pfthreshold'/>
<migration_features>
</migration_features>
<features>
</features>
<features>
</features>
root@op4:~# virsh capabilities |grep feature
<feature name='ht'/>
<feature name='monitor'/>
<feature name='osxsave'/>
<feature name='xsaves'/>
<feature name='cmp_legacy'/>
<feature name='extapic'/>
<feature name='skinit'/>
<feature name='wdt'/>
<feature name='tce'/>
<feature name='topoext'/>
<feature name='perfctr_core'/>
<feature name='perfctr_nb'/>
<feature name='invtsc'/>
<feature name='clzero'/>
<feature name='xsaveerptr'/>
<feature name='npt'/>
<feature name='lbrv'/>
<feature name='svm-lock'/>
<feature name='nrip-save'/>
<feature name='tsc-scale'/>
<feature name='vmcb-clean'/>
<feature name='flushbyasid'/>
<feature name='decodeassists'/>
<feature name='pause-filter'/>
<feature name='pfthreshold'/>
<migration_features>
</migration_features>
<features>
</features>
<features>
</features>
3) NUMA idem
root@op1:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
root@op4:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
) live-migration
openstack server list --all-project
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| 6b75b329-bda5-4348-a25e-ec1b9102b398 | u1 | ACTIVE | hub=xxxxxxxxxxxxx | N/A (booted from volume) | f1.mini |
#openstack server migrate --live-migration 6b75b329-bda5-4348-a25e-ec1b9102b398
No valid host was found. There are not enough hosts available. (HTTP 400) (Request-ID: req-a3489211-cb92-4744-a3b1-1df96bbcb3f2)
log from nova-cloud-controller (debug mode) here:
https://paste.ubuntu.com/p/kF3kmcXtQH/
where op4.maas is the source hypervisor and op1.maas is the destination
one: they have the same hardware, RAM and ceph storage.
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
charm bundle: openstack-2023.2
nova-compute: 28.0.1 channel 2023.2/stable
ubuntu: jammy .15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
I have two identical host, op1 and op4, DELL PowerEdge R7425 acting as nova-compute nodes (hypervisors).
ubuntu@juju:~/juju/controller$ openstack hypervisor list
+----+---------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+--------------+-------+
| 7 | op1.maas | QEMU | xxxxxxxxxxxx | up |
| 8 | op4.maas | QEMU | xxxxxxxxxxxx | up |
+----+---------------------+-----------------+--------------+-------+
-
1)microcode is the same:
root@op1:~# virsh capabilities |grep microcode
- <microcode version='134222446'/>
+ <microcode version='134222446'/>
root@op4:~# virsh capabilities |grep microcode
- <microcode version='134222446'/>
+ <microcode version='134222446'/>
2) libvirt capabilities features are identical:
root@op1:~# virsh capabilities |grep feature
- <feature name='ht'/>
- <feature name='monitor'/>
- <feature name='osxsave'/>
- <feature name='xsaves'/>
- <feature name='cmp_legacy'/>
- <feature name='extapic'/>
- <feature name='skinit'/>
- <feature name='wdt'/>
- <feature name='tce'/>
- <feature name='topoext'/>
- <feature name='perfctr_core'/>
- <feature name='perfctr_nb'/>
- <feature name='invtsc'/>
- <feature name='clzero'/>
- <feature name='xsaveerptr'/>
- <feature name='npt'/>
- <feature name='lbrv'/>
- <feature name='svm-lock'/>
- <feature name='nrip-save'/>
- <feature name='tsc-scale'/>
- <feature name='vmcb-clean'/>
- <feature name='flushbyasid'/>
- <feature name='decodeassists'/>
- <feature name='pause-filter'/>
- <feature name='pfthreshold'/>
- <migration_features>
- </migration_features>
- <features>
- </features>
- <features>
- </features>
+ <feature name='ht'/>
+ <feature name='monitor'/>
+ <feature name='osxsave'/>
+ <feature name='xsaves'/>
+ <feature name='cmp_legacy'/>
+ <feature name='extapic'/>
+ <feature name='skinit'/>
+ <feature name='wdt'/>
+ <feature name='tce'/>
+ <feature name='topoext'/>
+ <feature name='perfctr_core'/>
+ <feature name='perfctr_nb'/>
+ <feature name='invtsc'/>
+ <feature name='clzero'/>
+ <feature name='xsaveerptr'/>
+ <feature name='npt'/>
+ <feature name='lbrv'/>
+ <feature name='svm-lock'/>
+ <feature name='nrip-save'/>
+ <feature name='tsc-scale'/>
+ <feature name='vmcb-clean'/>
+ <feature name='flushbyasid'/>
+ <feature name='decodeassists'/>
+ <feature name='pause-filter'/>
+ <feature name='pfthreshold'/>
+ <migration_features>
+ </migration_features>
+ <features>
+ </features>
+ <features>
+ </features>
root@op4:~# virsh capabilities |grep feature
- <feature name='ht'/>
- <feature name='monitor'/>
- <feature name='osxsave'/>
- <feature name='xsaves'/>
- <feature name='cmp_legacy'/>
- <feature name='extapic'/>
- <feature name='skinit'/>
- <feature name='wdt'/>
- <feature name='tce'/>
- <feature name='topoext'/>
- <feature name='perfctr_core'/>
- <feature name='perfctr_nb'/>
- <feature name='invtsc'/>
- <feature name='clzero'/>
- <feature name='xsaveerptr'/>
- <feature name='npt'/>
- <feature name='lbrv'/>
- <feature name='svm-lock'/>
- <feature name='nrip-save'/>
- <feature name='tsc-scale'/>
- <feature name='vmcb-clean'/>
- <feature name='flushbyasid'/>
- <feature name='decodeassists'/>
- <feature name='pause-filter'/>
- <feature name='pfthreshold'/>
- <migration_features>
- </migration_features>
- <features>
- </features>
- <features>
- </features>
+ <feature name='ht'/>
+ <feature name='monitor'/>
+ <feature name='osxsave'/>
+ <feature name='xsaves'/>
+ <feature name='cmp_legacy'/>
+ <feature name='extapic'/>
+ <feature name='skinit'/>
+ <feature name='wdt'/>
+ <feature name='tce'/>
+ <feature name='topoext'/>
+ <feature name='perfctr_core'/>
+ <feature name='perfctr_nb'/>
+ <feature name='invtsc'/>
+ <feature name='clzero'/>
+ <feature name='xsaveerptr'/>
+ <feature name='npt'/>
+ <feature name='lbrv'/>
+ <feature name='svm-lock'/>
+ <feature name='nrip-save'/>
+ <feature name='tsc-scale'/>
+ <feature name='vmcb-clean'/>
+ <feature name='flushbyasid'/>
+ <feature name='decodeassists'/>
+ <feature name='pause-filter'/>
+ <feature name='pfthreshold'/>
+ <migration_features>
+ </migration_features>
+ <features>
+ </features>
+ <features>
+ </features>
3) NUMA idem
root@op1:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
root@op4:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
-
) live-migration
openstack server list --all-project
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| 6b75b329-bda5-4348-a25e-ec1b9102b398 | u1 | ACTIVE | hub=xxxxxxxxxxxxx | N/A (booted from volume) | f1.mini |
-
- #openstack server migrate --live-migration 6b75b329-bda5-4348-a25e-ec1b9102b398
+ #openstack server migrate --live-migration 6b75b329-bda5-4348-a25e-ec1b9102b398
+ No valid host was found. There are not enough hosts available. (HTTP 400) (Request-ID: req-a3489211-cb92-4744-a3b1-1df96bbcb3f2)
log from nova-cloud-controller (debug mode) here:
https://paste.ubuntu.com/p/kF3kmcXtQH/
where op4.maas is the source hypervisor and op1.maas is the destination
one: they have the same hardware, RAM and ceph storage.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2061569
Title:
Live migration fails despite identical CPUs with Host filter ignoring
hosts
Status in OpenStack Compute (nova):
New
Bug description:
charm bundle: openstack-2023.2
nova-compute: 28.0.1 channel 2023.2/stable
ubuntu: jammy .15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
I have two identical host, op1 and op4, DELL PowerEdge R7425 acting as nova-compute nodes (hypervisors).
ubuntu@juju:~/juju/controller$ openstack hypervisor list
+----+---------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+--------------+-------+
| 7 | op1.maas | QEMU | xxxxxxxxxxxx | up |
| 8 | op4.maas | QEMU | xxxxxxxxxxxx | up |
+----+---------------------+-----------------+--------------+-------+
1)microcode is the same:
root@op1:~# virsh capabilities |grep microcode
<microcode version='134222446'/>
root@op4:~# virsh capabilities |grep microcode
<microcode version='134222446'/>
2) libvirt capabilities features are identical:
root@op1:~# virsh capabilities |grep feature
<feature name='ht'/>
<feature name='monitor'/>
<feature name='osxsave'/>
<feature name='xsaves'/>
<feature name='cmp_legacy'/>
<feature name='extapic'/>
<feature name='skinit'/>
<feature name='wdt'/>
<feature name='tce'/>
<feature name='topoext'/>
<feature name='perfctr_core'/>
<feature name='perfctr_nb'/>
<feature name='invtsc'/>
<feature name='clzero'/>
<feature name='xsaveerptr'/>
<feature name='npt'/>
<feature name='lbrv'/>
<feature name='svm-lock'/>
<feature name='nrip-save'/>
<feature name='tsc-scale'/>
<feature name='vmcb-clean'/>
<feature name='flushbyasid'/>
<feature name='decodeassists'/>
<feature name='pause-filter'/>
<feature name='pfthreshold'/>
<migration_features>
</migration_features>
<features>
</features>
<features>
</features>
root@op4:~# virsh capabilities |grep feature
<feature name='ht'/>
<feature name='monitor'/>
<feature name='osxsave'/>
<feature name='xsaves'/>
<feature name='cmp_legacy'/>
<feature name='extapic'/>
<feature name='skinit'/>
<feature name='wdt'/>
<feature name='tce'/>
<feature name='topoext'/>
<feature name='perfctr_core'/>
<feature name='perfctr_nb'/>
<feature name='invtsc'/>
<feature name='clzero'/>
<feature name='xsaveerptr'/>
<feature name='npt'/>
<feature name='lbrv'/>
<feature name='svm-lock'/>
<feature name='nrip-save'/>
<feature name='tsc-scale'/>
<feature name='vmcb-clean'/>
<feature name='flushbyasid'/>
<feature name='decodeassists'/>
<feature name='pause-filter'/>
<feature name='pfthreshold'/>
<migration_features>
</migration_features>
<features>
</features>
<features>
</features>
3) NUMA idem
root@op1:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
root@op4:~# lscpu | grep -i numa
NUMA node(s): 8
NUMA node0 CPU(s): 0,8,16,24,32,40,48,56
NUMA node1 CPU(s): 2,10,18,26,34,42,50,58
NUMA node2 CPU(s): 4,12,20,28,36,44,52,60
NUMA node3 CPU(s): 6,14,22,30,38,46,54,62
NUMA node4 CPU(s): 1,9,17,25,33,41,49,57
NUMA node5 CPU(s): 3,11,19,27,35,43,51,59
NUMA node6 CPU(s): 5,13,21,29,37,45,53,61
NUMA node7 CPU(s): 7,15,23,31,39,47,55,63
) live-migration
openstack server list --all-project
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-------------------+--------------------------+---------+
| 6b75b329-bda5-4348-a25e-ec1b9102b398 | u1 | ACTIVE | hub=xxxxxxxxxxxxx | N/A (booted from volume) | f1.mini |
#openstack server migrate --live-migration 6b75b329-bda5-4348-a25e-ec1b9102b398
No valid host was found. There are not enough hosts available. (HTTP 400) (Request-ID: req-a3489211-cb92-4744-a3b1-1df96bbcb3f2)
log from nova-cloud-controller (debug mode) here:
https://paste.ubuntu.com/p/kF3kmcXtQH/
where op4.maas is the source hypervisor and op1.maas is the
destination one: they have the same hardware, RAM and ceph storage.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2061569/+subscriptions
