yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95942
[Bug 2112187] [NEW] Live migration fails after migrating the volume the instance has attached
Public bug reported:
Using Watcher, live migrating an instance that has a volume attached
fails after migrating the volume. The migration fails with
libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such
file or directory
Steps to reproduce the nova live migrate issue after volume migration:
1. Create an audit template:
openstack optimize audittemplate create zm_test hardware_maintenance --strategy zone_migration
2. Create vms with volumes:
set -ex
# Create Image
IMG=cirros-0.5.2-x86_64-disk.img
URL=http://download.cirros-cloud.net/0.5.2/$IMG
DISK_FORMAT=qcow2
RAW=$IMG
NUMBER_OF_INSTANCES=${1:-8}
# Create flavor
openstack flavor show m1.tiny || \
openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny
curl -L -# $URL > /tmp/$IMG
if type qemu-img >/dev/null 2>&1; then
RAW=$(echo $IMG | sed s/img/raw/g)
qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW
DISK_FORMAT=raw
fi
openstack image show cirros || \
openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW
# Create security group and icmp/ssh rules
openstack security group show basic || {
openstack security group create basic
openstack security group rule create basic --protocol icmp --ingress --icmp-type -1
openstack security group rule create basic --protocol tcp --ingress --dst-port 22
}
# Create an instance
for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do
NAME=test_${i}
VOL_NAME=tes_vol_${i}
# create a volume
openstack volume create --size ${i} --image cirros ${VOL_NAME}
openstack server show ${NAME} || {
openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait
fip=$(openstack floating ip create public -f value -c floating_ip_address)
openstack server add floating ip ${NAME} $fip
}
openstack server add volume ${NAME} ${VOL_NAME}
openstack server list --long
done
3. Create the audit:
openstack optimize audit create -a zm_test -p
storage_pools='[{"src_pool": "jgilaber-
watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber-
watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1",
"dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node": "jgilaber-
watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p
with_attached_volume=true
Note the audit id that is generated
4. Check the action plan generated
openstack optimize actionplan list --audit <audit_id>
Note the action plan uuid
5. Start the proposed action plan
openstack optimize actionplan start <action_plan_id>
As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed.
List of events:
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 |
| req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 |
| req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 |
| req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
```
```
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | swap_volume |
| events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= |
| id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| start_time | 2025-05-30T08:41:16.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | live-migration |
| events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= |
| | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= |
| | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= |
| | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= |
| id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| start_time | 2025-05-30T08:42:08.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-----------------------------------------------------------------------------------------------------------------------------
Compute-nova logs from the source host with the error and guest xml
https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2112187
Title:
Live migration fails after migrating the volume the instance has
attached
Status in OpenStack Compute (nova):
New
Bug description:
Using Watcher, live migrating an instance that has a volume attached
fails after migrating the volume. The migration fails with
libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such
file or directory
Steps to reproduce the nova live migrate issue after volume migration:
1. Create an audit template:
openstack optimize audittemplate create zm_test hardware_maintenance --strategy zone_migration
2. Create vms with volumes:
set -ex
# Create Image
IMG=cirros-0.5.2-x86_64-disk.img
URL=http://download.cirros-cloud.net/0.5.2/$IMG
DISK_FORMAT=qcow2
RAW=$IMG
NUMBER_OF_INSTANCES=${1:-8}
# Create flavor
openstack flavor show m1.tiny || \
openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny
curl -L -# $URL > /tmp/$IMG
if type qemu-img >/dev/null 2>&1; then
RAW=$(echo $IMG | sed s/img/raw/g)
qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW
DISK_FORMAT=raw
fi
openstack image show cirros || \
openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW
# Create security group and icmp/ssh rules
openstack security group show basic || {
openstack security group create basic
openstack security group rule create basic --protocol icmp --ingress --icmp-type -1
openstack security group rule create basic --protocol tcp --ingress --dst-port 22
}
# Create an instance
for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do
NAME=test_${i}
VOL_NAME=tes_vol_${i}
# create a volume
openstack volume create --size ${i} --image cirros ${VOL_NAME}
openstack server show ${NAME} || {
openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait
fip=$(openstack floating ip create public -f value -c floating_ip_address)
openstack server add floating ip ${NAME} $fip
}
openstack server add volume ${NAME} ${VOL_NAME}
openstack server list --long
done
3. Create the audit:
openstack optimize audit create -a zm_test -p
storage_pools='[{"src_pool": "jgilaber-
watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber-
watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1",
"dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node":
"jgilaber-watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p
with_attached_volume=true
Note the audit id that is generated
4. Check the action plan generated
openstack optimize actionplan list --audit <audit_id>
Note the action plan uuid
5. Start the proposed action plan
openstack optimize actionplan start <action_plan_id>
As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed.
List of events:
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
| req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 |
| req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 |
| req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 |
| req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 |
+------------------------------------------+--------------------------------------+----------------+----------------------------+
```
```
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | swap_volume |
| events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= |
| id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 |
| start_time | 2025-05-30T08:41:16.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | live-migration |
| events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= |
| | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= |
| | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', |
| | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= |
| | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= |
| | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', |
| | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= |
| id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| message | None |
| project_id | 14f6d4c19ca04da0b68cd09b986428a8 |
| request_id | req-c805b970-97cd-424d-8415-7c30f4101399 |
| start_time | 2025-05-30T08:42:08.000000 |
| user_id | af0c6c70cc2a46e1abe39d3ad31de18c |
+------------+-----------------------------------------------------------------------------------------------------------------------------
Compute-nova logs from the source host with the error and guest xml
https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2112187/+subscriptions
