| Thread Previous • Date Previous • Date Next • Thread Next |
Public bug reported:
Description
===========
After shut off an instance and starting it again the disk path changes inside the virtual instance. This is usually a small issue if you are only using the disk label/or any other ID rather than disk path.
If any application is using the disk path inside windows the application becamomes failed until replacing the device path.
Also I understand that most people won't even notice the problem, but we
don't have similar issues with VMware and we believe that the path
should be the same regardless of any kind of shut off/startup operation.
Steps to reproduce
==================
- Deploy a new instance - eg: Windows 2012 R2
- Attach a new CEPH block device volume
- Verify the disk path inside Windows
Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&1618751F&0&000000
Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&30
Location: Bus Number 0, Target Id 0, LUN 0
- Shut off the instance
- Start the instance
- The disk is offline after the startup, changing to online working
- Verify the disk path:
Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&352177D0&1&000000
Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&28
Location: Bus Number 0, Target Id 0, LUN 0
Using Ubuntu 16.04 LTS:
- Spin up a new Instance using Ubuntu 16.04 LTS
- Attach a new CEPH block device
root@attach-test-2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 16G 0 disk
└─vda1 253:1 0 16G 0 part /
vdb 253:16 0 1G 0 disk
root@attach-test-2:~# ls -la /dev/disk/by-path
total 0
drwxr-xr-x 2 root root 100 Jun 17 11:26 .
drwxr-xr-x 6 root root 120 Jun 17 11:26 ..
lrwxrwxrwx 1 root root 9 Jun 17 11:23 virtio-pci-0000:00:04.0 -> ../../vda
lrwxrwxrwx 1 root root 10 Jun 17 11:23 virtio-pci-0000:00:04.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root 9 Jun 17 11:26 virtio-pci-0000:00:06.0 -> ../../vdb
- Shut ioff the instance
- Start the instance
- Check disk path
root@attach-test-2:~# ls -la /dev/disk/by-path
total 0
drwxr-xr-x 2 root root 100 Jun 17 11:28 .
drwxr-xr-x 6 root root 120 Jun 17 11:28 ..
lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:04.0 -> ../../vda
lrwxrwxrwx 1 root root 10 Jun 17 11:28 virtio-pci-0000:00:04.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:05.0 -> ../../vdb
THe disk path changed using Linux too
I have a feeling that this is a KVM/LibVirt issue...
Expected result
===============
Nova & Libvirt & KVM should use the same details when attaching the disk on the instance regardless how many time we shut it down and start it.
So both instance path and parent should be unchanged.
Actual result
=============
Path changes
Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Hypervisors:
KVM + Libvirt on all compute nodes
Storage:
CEPH, iSCSI and FC using Netapp
CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library
Networking
Neutron networking with Openvswitch
----
root@openstack1:~# nova show Attach_Test_Win2k12
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hostname | attach-test-win2k12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000114 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-jiddtkfq |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-17T10:44:43.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-06-17T10:44:27Z |
| description | Attach_Test_Win2k12 |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| host_status | UP |
| id | 08c04ee9-b922-408d-8642-0e8f1fa59a1d |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| locked | False |
| metadata | {} |
| name | Attach_Test_Win2k12 |
| net-10.10.128 network | 10.10.128.51 |
| os-extended-volumes:volumes_attached | [{"id": "32595603-6c42-4bcb-818d-53b0c93b7c06", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | CCL |
| status | ACTIVE |
| tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| updated | 2016-06-17T10:46:15Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+----------------------------------------------------------------------------------+
root@openstack1:~# openstack volume show 32595603-6c42-4bcb-818d-53b0c93b7c06
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'08c04ee9-b922-408d-8642-0e8f1fa59a1d', u'attachment_id': u'280a5b3b-599b-41a9-9793-ea6d774a1cb7', u'host_name': None, u'volume_id': u'32595603-6c42-4bcb-818d-53b0c93b7c06', u'device': u'/dev/vdb', u'id': u'32595603-6c42-4bcb-818d-53b0c93b7c06'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-06-17T10:53:38.000000 |
| description | |
| encrypted | False |
| id | 32595603-6c42-4bcb-818d-53b0c93b7c06 |
| migration_status | None |
| multiattach | False |
| name | Attachment_Test_1 |
| os-vol-host-attr:host | openstackblock.ccl-dev.net@rbd#CEPH |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | CEPH |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
** Affects: nova
Importance: Undecided
Status: New
** Attachment added: "Libvirt.xml - After the volume attached. Nothing in the XML yet."
https://bugs.launchpad.net/bugs/1593684/+attachment/4685582/+files/libvirt_after_first_attach.xml
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1593684
Title:
Nova & Cinder - Disk attachement changes after shutoff/start
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
After shut off an instance and starting it again the disk path changes inside the virtual instance. This is usually a small issue if you are only using the disk label/or any other ID rather than disk path.
If any application is using the disk path inside windows the application becamomes failed until replacing the device path.
Also I understand that most people won't even notice the problem, but
we don't have similar issues with VMware and we believe that the path
should be the same regardless of any kind of shut off/startup
operation.
Steps to reproduce
==================
- Deploy a new instance - eg: Windows 2012 R2
- Attach a new CEPH block device volume
- Verify the disk path inside Windows
Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&1618751F&0&000000
Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&30
Location: Bus Number 0, Target Id 0, LUN 0
- Shut off the instance
- Start the instance
- The disk is offline after the startup, changing to online working
- Verify the disk path:
Device instance path: SCSI\DISK&VEN_RED_HAT&PROD_VIRTIO\4&352177D0&1&000000
Parent: PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13c0b0c5&0&28
Location: Bus Number 0, Target Id 0, LUN 0
Using Ubuntu 16.04 LTS:
- Spin up a new Instance using Ubuntu 16.04 LTS
- Attach a new CEPH block device
root@attach-test-2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 16G 0 disk
└─vda1 253:1 0 16G 0 part /
vdb 253:16 0 1G 0 disk
root@attach-test-2:~# ls -la /dev/disk/by-path
total 0
drwxr-xr-x 2 root root 100 Jun 17 11:26 .
drwxr-xr-x 6 root root 120 Jun 17 11:26 ..
lrwxrwxrwx 1 root root 9 Jun 17 11:23 virtio-pci-0000:00:04.0 -> ../../vda
lrwxrwxrwx 1 root root 10 Jun 17 11:23 virtio-pci-0000:00:04.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root 9 Jun 17 11:26 virtio-pci-0000:00:06.0 -> ../../vdb
- Shut ioff the instance
- Start the instance
- Check disk path
root@attach-test-2:~# ls -la /dev/disk/by-path
total 0
drwxr-xr-x 2 root root 100 Jun 17 11:28 .
drwxr-xr-x 6 root root 120 Jun 17 11:28 ..
lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:04.0 -> ../../vda
lrwxrwxrwx 1 root root 10 Jun 17 11:28 virtio-pci-0000:00:04.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root 9 Jun 17 11:28 virtio-pci-0000:00:05.0 -> ../../vdb
THe disk path changed using Linux too
I have a feeling that this is a KVM/LibVirt issue...
Expected result
===============
Nova & Libvirt & KVM should use the same details when attaching the disk on the instance regardless how many time we shut it down and start it.
So both instance path and parent should be unchanged.
Actual result
=============
Path changes
Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
Hypervisors:
KVM + Libvirt on all compute nodes
Storage:
CEPH, iSCSI and FC using Netapp
CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library
Networking
Neutron networking with Openvswitch
----
root@openstack1:~# nova show Attach_Test_Win2k12
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hostname | attach-test-win2k12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000114 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-jiddtkfq |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-17T10:44:43.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-06-17T10:44:27Z |
| description | Attach_Test_Win2k12 |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| host_status | UP |
| id | 08c04ee9-b922-408d-8642-0e8f1fa59a1d |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| locked | False |
| metadata | {} |
| name | Attach_Test_Win2k12 |
| net-10.10.128 network | 10.10.128.51 |
| os-extended-volumes:volumes_attached | [{"id": "32595603-6c42-4bcb-818d-53b0c93b7c06", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | CCL |
| status | ACTIVE |
| tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| updated | 2016-06-17T10:46:15Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+----------------------------------------------------------------------------------+
root@openstack1:~# openstack volume show 32595603-6c42-4bcb-818d-53b0c93b7c06
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'08c04ee9-b922-408d-8642-0e8f1fa59a1d', u'attachment_id': u'280a5b3b-599b-41a9-9793-ea6d774a1cb7', u'host_name': None, u'volume_id': u'32595603-6c42-4bcb-818d-53b0c93b7c06', u'device': u'/dev/vdb', u'id': u'32595603-6c42-4bcb-818d-53b0c93b7c06'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-06-17T10:53:38.000000 |
| description | |
| encrypted | False |
| id | 32595603-6c42-4bcb-818d-53b0c93b7c06 |
| migration_status | None |
| multiattach | False |
| name | Attachment_Test_1 |
| os-vol-host-attr:host | openstackblock.ccl-dev.net@rbd#CEPH |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | CEPH |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1593684/+subscriptions
| Thread Previous • Date Previous • Date Next • Thread Next |
