yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #25063
[Bug 871278] Re: Cannot attach volumes to instances if tgt is used
oneiric has seen the end of its life and is no longer receiving any
updates. Marking the oneiric task for this ticket as "Won't Fix".
** Changed in: nova (Ubuntu Oneiric)
Status: Incomplete => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/871278
Title:
Cannot attach volumes to instances if tgt is used
Status in OpenStack Compute (Nova):
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Oneiric:
Won't Fix
Status in nova source package in Precise:
Fix Released
Bug description:
I cannot attach a volume to an instance if both are on different
hosts.
This is a problem which is specific to tgt which is used on Oneiric. I
file that bug in Nova (Ubuntu) and not in Nova (Openstack), even
though it is not an Ubuntu packaging problem.
Example:
SW: OpenStack 2011.3 (Diablo) -- Package: 2011.3-0ubuntu6
OS: Ubuntu 11.10 (Oneiric -- Beta2)
node1 (192.168.1.201): vol-00000004
node2 (192.168.1.202): i-00000004
root@node1:~# euca-attach-volume -i i-00000004 -d /dev/vdc
vol-00000004
Error seen in nova-compute.log:
Error: iSCSI device not found at /dev/disk/by-
path/ip-192.168.1.201:3260-iscsi-
iqn.2010-10.org.openstack:volume-00000004-lun-0
Explanation:
nova-compute is waiting for a device to be configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-
iqn.2010-10.org.openstack:volume-00000004-lun-0
but the device is configured as:
/dev/disk/by-path/ip-192.168.1.201:3260-iscsi-
iqn.2010-10.org.openstack:volume-00000004-lun-1
nova-compute waits for xxx-lun-0, but xxx-lun-1 is configured:
root@node2:/dev/disk/by-path# ls -l
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
I guess this is a difference between the iet and tgt softwares used to
configure iSCSI targets.
On Oneiric, tgt is used by default, and iet cannot be used because the
kernel module is no more available.
In nova.conf:
--iscsi_helper=tgtadm
If I adapt the code from lun-0 to lun-1, everything works well:
diff /usr/share/pyshared/nova/volume/driver.py-patched /usr/share/pyshared/nova/volume/driver.py-unpatched
536c536
< mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-1" %
---
> mount_device = ("/dev/disk/by-path/ip-%s-iscsi-%s-lun-0" %
Complete nova-compute.log:
root@node2:~# tail -50 /var/log/nova/nova-compute.log
2011-10-08 22:02:31,179 DEBUG nova.rpc [-] received {u'_context_roles': [u'projectmanager'], u'_context_request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', u'_context_read_deleted': False, u'args': {u'instance_id': 4, u'mountpoint': u'/dev/vdc', u'volume_id': 4}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'project-one', u'_context_timestamp': u'2011-10-08T20:02:31.145628', u'_context_user_id': u'admin', u'method': u'attach_volume', u'_context_remote_address': u'192.168.1.201'} from (pid=1585) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600
2011-10-08 22:02:31,180 DEBUG nova.rpc [-] unpacked context: {'user_id': u'admin', 'roles': [u'projectmanager'], 'timestamp': u'2011-10-08T20:02:31.145628', 'auth_token': None, 'msg_id': None, 'remote_address': u'192.168.1.201', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8911d1d6-66eb-48eb-ab9a-f7a387d858fa', 'project_id': u'project-one', 'read_deleted': False} from (pid=1585) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: decorating: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,181 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0x21e5790>| |<nova.rpc.impl_kombu.RpcContext object at 0x43fdf10>| |4|
2011-10-08 22:02:31,181 DEBUG nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: getting locked state from (pid=1585) get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1165
2011-10-08 22:02:31,333 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: locked: |False|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: admin: |True|
2011-10-08 22:02:31,334 INFO nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] check_instance_lock: executing: |<function attach_volume at 0x2946230>|
2011-10-08 22:02:31,410 AUDIT nova.compute.manager [8911d1d6-66eb-48eb-ab9a-f7a387d858fa admin project-one] instance 4: attaching volume 4 to /dev/vdc
2011-10-08 22:02:31,447 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-10-08 22:02:31,447 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p node1 from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:31,489 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 192.168.1.201:3260,1 iqn.2010-10.org.openstack:volume-00000004 from (pid=1585) _get_iscsi_properties /usr/lib/python2.7/dist-packages/nova/volume/driver.py:479
2011-10-08 22:02:31,490 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --login from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,015 DEBUG nova.volume.driver [-] iscsiadm ('--login',): stdout=Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]: successful
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,015 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --op update -n node.startup -v automatic from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,033 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'): stdout= stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:32,033 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 0
2011-10-08 22:02:32,034 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:32,052 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:33,054 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 1
2011-10-08 22:02:33,054 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:33,073 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:37,077 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0. Will rescan & retry. Try number: 2
2011-10-08 22:02:37,077 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000004 -p 192.168.1.201:3260 --rescan from (pid=1585) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-10-08 22:02:37,095 DEBUG nova.volume.driver [-] iscsiadm ('--rescan',): stdout=Rescanning session [sid: 3, target: iqn.2010-10.org.openstack:volume-00000004, portal: 192.168.1.201,3260]
stderr= from (pid=1585) _run_iscsiadm /usr/lib/python2.7/dist-packages/nova/volume/driver.py:506
2011-10-08 22:02:46,100 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1242, in attach_volume
(nova.rpc): TRACE: volume_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 245, in setup_compute_volume
(nova.rpc): TRACE: path = self.driver.discover_volume(context, volume_ref)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/driver.py", line 546, in discover_volume
(nova.rpc): TRACE: (mount_device))
(nova.rpc): TRACE: Error: iSCSI device not found at /dev/disk/by-path/ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-0
(nova.rpc): TRACE:
2011-10-08 22:03:52,889 INFO nova.compute.manager [-] Updating host status
2011-10-08 22:05:55,440 INFO nova.compute.manager [-] Updating host status
root@node2:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 9 2011-10-08 22:02 ip-192.168.1.201:3260-iscsi-iqn.2010-10.org.openstack:volume-00000004-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:0:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2011-10-08 21:38 pci-0000:00:1f.2-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 2011-10-08 20:35 pci-0000:00:1f.2-scsi-1:0:1:0 -> ../../sr0
root@node1:~# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2010-10.org.openstack:volume-00000004
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1993-08.org.debian:01:63503b146b7
Connection: 0
IP Address: 192.168.1.202 <========= initiator (i-00000004) on node2 is connected to target (vol-00000004) on node1
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1 <==================================== LUN = 1 (and not 0)
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/nova-volumes/volume-00000004
Backing store flags:
Account information:
ACL information:
ALL
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/871278/+subscriptions
