yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2044331] [NEW] Allocation audit error during live migration

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Haidong Pang <2044331@xxxxxxxxxxxxxxxxxx>
Date: Thu, 23 Nov 2023 06:43:10 -0000
Reply-to: Bug 2044331 <2044331@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========
In live migration, 
creates destination node allocation which can get accidentally deleted by audit.

Steps to reproduce
==================
()[root@busybox-openstack-db97db44f-vddg8 /]# openstack server list --all-project --long
+--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                | Image Name | Image ID | Availability Zone | Host              | Properties |
+--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+
| 31af6b71-df56-4f56-87fb-64d75d321285 | test-vm | ACTIVE | None       | Running     | share_net=192.168.111.9 |            |          | default-az        | node-6.domain.tld |            |
+--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+


()[root@busybox-openstack-db97db44f-vddg8 /]# openstack compute service list --service nova-compute
+-----+--------------+-------------------+------------+---------+-------+----------------------------+
|  ID | Binary       | Host              | Zone       | Status  | State | Updated At                 |
+-----+--------------+-------------------+------------+---------+-------+----------------------------+
| 133 | nova-compute | node-3.domain.tld | default-az | enabled | up    | 2023-11-23T06:22:15.000000 |
| 148 | nova-compute | node-1.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
| 151 | nova-compute | node-2.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
| 226 | nova-compute | node-6.domain.tld | default-az | enabled | up    | 2023-11-23T06:22:15.000000 |
| 587 | nova-compute | node-7.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
+-----+--------------+-------------------+------------+---------+-------+----------------------------+


()[root@busybox-openstack-db97db44f-vddg8 /]# openstack server migrate 31af6b71-df56-4f56-87fb-64d75d321285 --live node-3.domain.tld


()[root@nova-maintenance-77f7cf548f-p6rrv /]# nova-manage placement audit --verbose --delete
Deprecated: Option "notification_format" from group "DEFAULT" is deprecated. Use option "notification_format" from group "notifications".
Allocations were set against consumer UUID 31af6b71-df56-4f56-87fb-64d75d321285 but no existing instances or active migrations are related.
2023-11-23 14:23:49.356 144 INFO nova.scheduler.client.report [req-99bf4509-367b-480c-97d9-0c99676a93be - - - - -] Deleted allocation for unknown 31af6b71-df56-4f56-87fb-64d75d321285
Deleted allocations for consumer UUID 31af6b71-df56-4f56-87fb-64d75d321285 on Resource Provider 301d0960-9bc4-4a88-9860-44446286b525: {'MEMORY_MB': 1024, 'VCPU': 1}
Processed 1 allocation.

Expected result
===============
`nova-manage placement audit` can detect active migrations and skip them.

Actual result
=============
`nova-manage placement audit` considers the allocation created on the target node during live migration as an orphaned allocation and will delete it.（The reason for this is that the nova database currently lists the original node as the host for this virtual machine instance. This record will be updated to reflect the target node only upon the completion of the live migration process.）

Environment
===========
1. Exact version of OpenStack you are running. See the following
   Openstack Wallaby

2. Which hypervisor did you use?
   Libvirt + KVM

2. Which storage type did you use?
   Ceph

3. Which networking type did you use?
   Neutron

** Affects: nova
     Importance: Undecided
     Assignee: Haidong Pang (haidong-pang)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2044331

Title:
  Allocation audit error during live migration

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  In live migration, 
  creates destination node allocation which can get accidentally deleted by audit.

  Steps to reproduce
  ==================
  ()[root@busybox-openstack-db97db44f-vddg8 /]# openstack server list --all-project --long
  +--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+
  | ID                                   | Name    | Status | Task State | Power State | Networks                | Image Name | Image ID | Availability Zone | Host              | Properties |
  +--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+
  | 31af6b71-df56-4f56-87fb-64d75d321285 | test-vm | ACTIVE | None       | Running     | share_net=192.168.111.9 |            |          | default-az        | node-6.domain.tld |            |
  +--------------------------------------+---------+--------+------------+-------------+-------------------------+------------+----------+-------------------+-------------------+------------+

  
  ()[root@busybox-openstack-db97db44f-vddg8 /]# openstack compute service list --service nova-compute
  +-----+--------------+-------------------+------------+---------+-------+----------------------------+
  |  ID | Binary       | Host              | Zone       | Status  | State | Updated At                 |
  +-----+--------------+-------------------+------------+---------+-------+----------------------------+
  | 133 | nova-compute | node-3.domain.tld | default-az | enabled | up    | 2023-11-23T06:22:15.000000 |
  | 148 | nova-compute | node-1.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
  | 151 | nova-compute | node-2.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
  | 226 | nova-compute | node-6.domain.tld | default-az | enabled | up    | 2023-11-23T06:22:15.000000 |
  | 587 | nova-compute | node-7.domain.tld | ddd        | enabled | up    | 2023-11-23T06:22:15.000000 |
  +-----+--------------+-------------------+------------+---------+-------+----------------------------+

  
  ()[root@busybox-openstack-db97db44f-vddg8 /]# openstack server migrate 31af6b71-df56-4f56-87fb-64d75d321285 --live node-3.domain.tld

  
  ()[root@nova-maintenance-77f7cf548f-p6rrv /]# nova-manage placement audit --verbose --delete
  Deprecated: Option "notification_format" from group "DEFAULT" is deprecated. Use option "notification_format" from group "notifications".
  Allocations were set against consumer UUID 31af6b71-df56-4f56-87fb-64d75d321285 but no existing instances or active migrations are related.
  2023-11-23 14:23:49.356 144 INFO nova.scheduler.client.report [req-99bf4509-367b-480c-97d9-0c99676a93be - - - - -] Deleted allocation for unknown 31af6b71-df56-4f56-87fb-64d75d321285
  Deleted allocations for consumer UUID 31af6b71-df56-4f56-87fb-64d75d321285 on Resource Provider 301d0960-9bc4-4a88-9860-44446286b525: {'MEMORY_MB': 1024, 'VCPU': 1}
  Processed 1 allocation.

  Expected result
  ===============
  `nova-manage placement audit` can detect active migrations and skip them.

  Actual result
  =============
  `nova-manage placement audit` considers the allocation created on the target node during live migration as an orphaned allocation and will delete it.（The reason for this is that the nova database currently lists the original node as the host for this virtual machine instance. This record will be updated to reflect the target node only upon the completion of the live migration process.）

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
     Openstack Wallaby

  2. Which hypervisor did you use?
     Libvirt + KVM

  2. Which storage type did you use?
     Ceph

  3. Which networking type did you use?
     Neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2044331/+subscriptions