yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #32658
[Bug 1452032] [NEW] Device descriptor not removed with different iqn and multipath enabled.
Public bug reported:
[Environment]
OpenStack Kilo
Trusty 14.04.4
[Description]
if the attached multipath devices doesn't have same iqn like regular
lvm+iscsi backend, in_use will be false.
In that case,_disconnect_volume_multipath_iscsi() returns without
calling _remove_multipath_device_descriptor().
[Reproduction]
1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute
nodes:
iscsi_use_multipath = True
3) Create 3 cinder volumes
$ cinder create 1
$ cinder create 1
$ cinder create 1
$ cinder list
ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
4) Attach them to nova
$ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7
$ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291
$ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
5) Check on the nova-compute unit for the current multipath/session
status
tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7
tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291
tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
Multipath:
root@juju-1374999-machine-10:/home/ubuntu# multipath -ll
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 8:0:0:1 sdg 8:96 active ready running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 9:0:0:1 sda 8:0 active ready running
6) Detach the current volumes.
First.
ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach
b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e-
e1a31e2ba6e7
ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll"
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 9:0:0:1 sda 8:0 active ready running
Second raises the faulty state
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
Third, raises the faulty state also
sudo: unable to resolve host juju-1374999-machine-10
33000000300000001 dm-2 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
Manual fix.
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001
** Affects: nova
Importance: Undecided
Assignee: Jorge Niedbalski (niedbalski)
Status: In Progress
** Affects: ubuntu
Importance: Undecided
Status: New
** Changed in: nova
Status: New => In Progress
** Changed in: nova
Assignee: (unassigned) => Jorge Niedbalski (niedbalski)
** Description changed:
[Environment]
OpenStack Kilo
Trusty 14.04.4
[Description]
if the attached multipath devices doesn't have same iqn like regular
lvm+iscsi backend, in_use will be false.
In that case,_disconnect_volume_multipath_iscsi() returns without
calling _remove_multipath_device_descriptor().
[Reproduction]
1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
-
- 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute nodes:
+ 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute
+ nodes:
iscsi_use_multipath = True
-
- 3) Create 3 cinder volumes
+ 3) Create 3 cinder volumes
$ cinder create 1
$ cinder create 1
$ cinder create 1
$ cinder list
ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
-
4) Attach them to nova
$ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7
$ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291
- $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
+ $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
5) Check on the nova-compute unit for the current multipath/session
status
tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7
tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291
tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
Multipath:
root@juju-1374999-machine-10:/home/ubuntu# multipath -ll
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 10:0:0:1 sdb 8:16 active ready running
+ `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 8:0:0:1 sdg 8:96 active ready running
+ `- 8:0:0:1 sdg 8:96 active ready running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 9:0:0:1 sda 8:0 active ready running
+ `- 9:0:0:1 sda 8:0 active ready running
6) Detach the current volumes.
First.
ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach
b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e-
e1a31e2ba6e7
-
ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll"
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 10:0:0:1 sdb 8:16 active ready running
+ `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 9:0:0:1 sda 8:0 active ready running
+ `- 9:0:0:1 sda 8:0 active ready running
Second raises the faulty state
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
- `- 10:0:0:1 sdb 8:16 active ready running
+ `- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
Third, raises the faulty state also
sudo: unable to resolve host juju-1374999-machine-10
33000000300000001 dm-2 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
- `- #:#:#:# - #:# active faulty running
+ `- #:#:#:# - #:# active faulty running
+
+
+ Manual fix.
+
+ root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001
+ root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001
+ root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1452032
Title:
Device descriptor not removed with different iqn and multipath
enabled.
Status in OpenStack Compute (Nova):
In Progress
Status in Ubuntu:
New
Bug description:
[Environment]
OpenStack Kilo
Trusty 14.04.4
[Description]
if the attached multipath devices doesn't have same iqn like regular
lvm+iscsi backend, in_use will be false.
In that case,_disconnect_volume_multipath_iscsi() returns without
calling _remove_multipath_device_descriptor().
[Reproduction]
1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute
nodes:
iscsi_use_multipath = True
3) Create 3 cinder volumes
$ cinder create 1
$ cinder create 1
$ cinder create 1
$ cinder list
ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
| 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
| 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 |
+--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+
4) Attach them to nova
$ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7
$ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291
$ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
5) Check on the nova-compute unit for the current multipath/session
status
tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7
tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291
tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f
Multipath:
root@juju-1374999-machine-10:/home/ubuntu# multipath -ll
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 8:0:0:1 sdg 8:96 active ready running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 9:0:0:1 sda 8:0 active ready running
6) Detach the current volumes.
First.
ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach
b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e-
e1a31e2ba6e7
ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll"
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 9:0:0:1 sda 8:0 active ready running
Second raises the faulty state
33000000300000001 dm-2 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:1 sdb 8:16 active ready running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
Third, raises the faulty state also
sudo: unable to resolve host juju-1374999-machine-10
33000000300000001 dm-2 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000100000001 dm-0 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
33000000200000001 dm-1 ,
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
`- #:#:#:# - #:# active faulty running
Manual fix.
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001
root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1452032/+subscriptions
Follow ups
References
