yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #96316
[Bug 2112187] Re: Live migration fails after migrating the volume the instance has attached
Reviewed: https://review.opendev.org/c/openstack/watcher/+/957768
Committed: https://opendev.org/openstack/watcher/commit/3742e0a79cfcbd039cd396132a4188f376fcfeca
Submitter: "Zuul (22348)"
Branch: master
commit 3742e0a79cfcbd039cd396132a4188f376fcfeca
Author: Sean Mooney <work@xxxxxxxxxxxxxxx>
Date: Tue Jun 10 20:13:49 2025 +0100
use cinder migrate for swap volume
This change removes watchers in tree functionality
for swapping instance volumes and defines swap as an alias
of cinder volume migrate.
The watcher native implementation was missing error handling
which could lead to irretrievable data loss.
The removed code also forged project user credentials to
perform admin request as if it was done by a member of a project.
this was unsafe an posses a security risk due to how it was
implemented. This code has been removed without replacement.
While some effort has been made to allow existing
audits that were defined to work, any reduction of functionality
as a result of this security hardening is intentional.
Closes-Bug: #2112187
Change-Id: Ic3b6bfd164e272d70fe86d7b182478dd962f8ac0
Signed-off-by: Sean Mooney <work@xxxxxxxxxxxxxxx>
** Changed in: watcher
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2112187
Title:
Live migration fails after migrating the volume the instance has
attached
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Security Advisory:
Won't Fix
Status in OpenStack Security Notes:
In Progress
Status in tempest:
In Progress
Status in watcher:
Fix Released
Bug description:
Using Watcher, live migrating an instance that has a volume attached
fails after migrating the volume. The migration fails with
libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such
file or directory
Steps to reproduce the nova live migrate issue after volume migration:
1. Create an audit template:
openstack optimize audittemplate create zm_test hardware_maintenance
--strategy zone_migration
2. Create vms with volumes:
set -ex
# Create Image
IMG=cirros-0.5.2-x86_64-disk.img
URL=http://download.cirros-cloud.net/0.5.2/$IMG
DISK_FORMAT=qcow2
RAW=$IMG
NUMBER_OF_INSTANCES=${1:-8}
# Create flavor
openstack flavor show m1.tiny || \
openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny
curl -L -# $URL > /tmp/$IMG
if type qemu-img >/dev/null 2>&1; then
RAW=$(echo $IMG | sed s/img/raw/g)
qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW
DISK_FORMAT=raw
fi
openstack image show cirros || \
openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW
# Create security group and icmp/ssh rules
openstack security group show basic || {
openstack security group create basic
openstack security group rule create basic --protocol icmp --ingress --icmp-type -1
openstack security group rule create basic --protocol tcp --ingress --dst-port 22
}
# Create an instance
for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do
NAME=test_${i}
VOL_NAME=tes_vol_${i}
# create a volume
openstack volume create --size ${i} --image cirros ${VOL_NAME}
openstack server show ${NAME} || {
openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait
fip=$(openstack floating ip create public -f value -c floating_ip_address)
openstack server add floating ip ${NAME} $fip
}
openstack server add volume ${NAME} ${VOL_NAME}
openstack server list --long
done
3. Create the audit:
openstack optimize audit create -a zm_test -p
storage_pools='[{"src_pool": "jgilaber-
watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber-
watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1",
"dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node":
"jgilaber-watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p
with_attached_volume=true
Note the audit id that is generated
4. Check the action plan generated
openstack optimize actionplan list --audit <audit_id>
Note the action plan uuid
5. Start the proposed action plan
openstack optimize actionplan start <action_plan_id>
As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed.
List of events:
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 |
| req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 |
| req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 |
| req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
```
```
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | swap_volume |
| events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= |
| id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| start_time | 2025-05-30T08:41:16.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | live-migration |
| events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= |
| | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= |
| | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= |
| | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= |
| id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| start_time | 2025-05-30T08:42:08.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-----------------------------------------------------------------------------------------------------------------------------
Compute-nova logs from the source host with the error and guest xml
https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2112187/+subscriptions
References
