yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1408329] [NEW] Error attaching volumes to instances on XenServer

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: nedo <bhnedo@xxxxxxxxxxx>
Date: Wed, 07 Jan 2015 15:02:47 -0000
Reply-to: Bug 1408329 <1408329@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am not able to attach Cinder Juno volumes to instances inside
XenServer. My storage node is a PV guest running inside XenServer with
200 GB cinder-volumes VG, and with tgt service configured as follow:

/etc/tgt/targets.conf

include /etc/tgt/conf.d/cinder.conf
# Set the driver. If not specified, defaults to "iscsi".
default-driver iscsi

/etc/tgt/conf.d/cinder.conf

include /etc/cinder/volumes/*

The volume is created successfully as I can see it from Horizon
dashboard, and also the LUN from tgtadm output:

tgtadm --op show --mode target

Target 2: iqn.2010-10.org.openstack:volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00020000
            SCSI SN: beaf20
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00020001
            SCSI SN: beaf21
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder-volumes/volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
            Backing store flags:
    Account information:
        a4ym7KcCR7TZ6sLj8sgm
    ACL information:
        ALL

When I try to attach the volume to an instance, the following error is
thrown on compute node:

nova-manage.log

WARNING nova.volume.cinder [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
2015-01-07 15:42:56.464 2114 WARNING nova.virt.xenapi.volume_utils [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Ignoring exception ['VDI_IN_USE', 'OpaqueRef:2225b8fb-0a06-f859-0648-09f71aee330e'] when unplugging PBD OpaqueRef:54856f0b-bcca-a114-c96c-a86a528d3761
2015-01-07 15:42:56.507 2114 ERROR root [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 67, in _attach_volume\n    dev_number, hotplug)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 124, in _attach_volume_to_vm\n    self._session.VBD.plug(vbd_ref, vm_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 84, in plug\n    synchronized_plug()\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 79, in synchronized_plug\n    self._call_method("plug", vbd_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 59, in _call_method\n    return self.session.call_xenapi(call, *args)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi\n    return session.xenapi_request(method, args)\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request\n    result = _parse_result(getattr(self, methodname)(*full_params))\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result\n    raise Failure(result[\'ErrorDescription\'])\n', "Failure: ['SR_BACKEND_FAILURE_46', '', 'The VDI is not available', '']\n"]
2015-01-07 15:42:56.508 2114 ERROR nova.virt.block_device [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Driver failed to attach volume bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
2015-01-07 15:42:57.167 2114 ERROR nova.compute.manager [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failed to attach bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     do_check_attach=False, do_driver_attach=True)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     ret_val = method(obj, context, *args, **kwargs)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     connector)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
2015-01-07 15:42:59.292 2114 ERROR oslo.messaging.rpc.dispatcher [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Exception during message handling: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 414, in decorated_function
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     payload)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     pass
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 284, in decorated_function
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4549, in attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_attach_volume(context, instance, driver_bdm)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4547, in do_attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     bdm.destroy(context)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4544, in do_attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._attach_volume(context, instance, driver_bdm)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4567, in _attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, bdm.volume_id)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     connector)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     mountpoint)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     instance_name, dev_number, hotplug)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     volume_utils.forget_sr(self._session, sr_ref)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     session.call_xenapi("SR.forget", sr_ref)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return session.xenapi_request(method, args)
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = _parse_result(getattr(self, methodname)(*full_params))
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     raise Failure(result['ErrorDescription'])
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher

Nova config file: /etc/nova/nova.conf

[DEFAULT]
rpc_backend = qpid
auth_strategy = keystone

network_api_class = nova.network.api.API
security_group_api = nova
network_manager = nova.network.manager.FlatDHCPManager
allow_same_net_traffic = True
multi_host = True
send_arp_for_ha = True
share_dhcp_address = True
force_dhcp_release = True
flat_network_bridge = xenbr0
flat_interface = eth1
public_interface = eth0
volume_api_class=nova.volume.cinder.API

firewall_driver = nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
volume_api_class=nova.volume.cinder.API
debug = True
verbose = True

qpid_hostname=192.168.1.7
qpid_port=5672

Cinder config file: /etc/cinder/cinder.conf

[DEFAULT]
rpc_backend = qpid
auth_strategy = keystone
my_ip = 192.168.1.107
glance_host = 192.168.1.7
iscsi_helper = tgtadm
verbose = True

enabled_backends=lvm

volume.log

2015-01-07 15:42:04.726 INFO cinder.brick.iscsi.iscsi [req-1b493d4d-18d7-4be9-9c23-0edf17a2c9ce 4389a9a7146844c0806ed9b756c69b0d e303a4d222b04e9f9217d1366eb8d945 - - -] Creating iscsi_target for: volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
2015-01-07 15:42:04.794 2162 INFO cinder.volume.manager [-] Updating volume status
2015-01-07 15:42:05.048 2162 INFO cinder.volume.manager [-] Updating volume replication status.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  I am not able to attach Cinder volumes to instances inside XenServer. My
- storage node is a PV guest inside XenServer with 200 GB cinder-volumes
- VG, and with tgt service configured as follow:
+ storage node is a PV guest running inside XenServer with 200 GB cinder-
+ volumes VG, and with tgt service configured as follow:
  
  /etc/tgt/targets.conf
  
  include /etc/tgt/conf.d/cinder.conf
  # Set the driver. If not specified, defaults to "iscsi".
  default-driver iscsi
  
  /etc/tgt/conf.d/cinder.conf
  
  include /etc/cinder/volumes/*
  
  The volume is created successfully as I can see it from Horizon
  dashboard, and also the LUN from tgtadm output:
  
- 
  tgtadm --op show --mode target
  
  Target 2: iqn.2010-10.org.openstack:volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
-     System information:
-         Driver: iscsi
-         State: ready
-     I_T nexus information:
-     LUN information:
-         LUN: 0
-             Type: controller
-             SCSI ID: IET     00020000
-             SCSI SN: beaf20
-             Size: 0 MB, Block size: 1
-             Online: Yes
-             Removable media: No
-             Prevent removal: No
-             Readonly: No
-             SWP: No
-             Thin-provisioning: No
-             Backing store type: null
-             Backing store path: None
-             Backing store flags:
-         LUN: 1
-             Type: disk
-             SCSI ID: IET     00020001
-             SCSI SN: beaf21
-             Size: 1074 MB, Block size: 512
-             Online: Yes
-             Removable media: No
-             Prevent removal: No
-             Readonly: No
-             SWP: No
-             Thin-provisioning: No
-             Backing store type: rdwr
-             Backing store path: /dev/cinder-volumes/volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
-             Backing store flags:
-     Account information:
-         a4ym7KcCR7TZ6sLj8sgm
-     ACL information:
-         ALL
+     System information:
+         Driver: iscsi
+         State: ready
+     I_T nexus information:
+     LUN information:
+         LUN: 0
+             Type: controller
+             SCSI ID: IET     00020000
+             SCSI SN: beaf20
+             Size: 0 MB, Block size: 1
+             Online: Yes
+             Removable media: No
+             Prevent removal: No
+             Readonly: No
+             SWP: No
+             Thin-provisioning: No
+             Backing store type: null
+             Backing store path: None
+             Backing store flags:
+         LUN: 1
+             Type: disk
+             SCSI ID: IET     00020001
+             SCSI SN: beaf21
+             Size: 1074 MB, Block size: 512
+             Online: Yes
+             Removable media: No
+             Prevent removal: No
+             Readonly: No
+             SWP: No
+             Thin-provisioning: No
+             Backing store type: rdwr
+             Backing store path: /dev/cinder-volumes/volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
+             Backing store flags:
+     Account information:
+         a4ym7KcCR7TZ6sLj8sgm
+     ACL information:
+         ALL
  
  When I try to attach the volume to an instance, the following error is
  thrown on compute node:
  
  nova-manage.log
- 
  
  WARNING nova.volume.cinder [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
  2015-01-07 15:42:56.464 2114 WARNING nova.virt.xenapi.volume_utils [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Ignoring exception ['VDI_IN_USE', 'OpaqueRef:2225b8fb-0a06-f859-0648-09f71aee330e'] when unplugging PBD OpaqueRef:54856f0b-bcca-a114-c96c-a86a528d3761
  2015-01-07 15:42:56.507 2114 ERROR root [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 67, in _attach_volume\n    dev_number, hotplug)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 124, in _attach_volume_to_vm\n    self._session.VBD.plug(vbd_ref, vm_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 84, in plug\n    synchronized_plug()\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 79, in synchronized_plug\n    self._call_method("plug", vbd_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 59, in _call_method\n    return self.session.call_xenapi(call, *args)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi\n    return session.xenapi_request(method, args)\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request\n    result = _parse_result(getattr(self, methodname)(*full_params))\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result\n    raise Failure(result[\'ErrorDescription\'])\n', "Failure: ['SR_BACKEND_FAILURE_46', '', 'The VDI is not available', '']\n"]
  2015-01-07 15:42:56.508 2114 ERROR nova.virt.block_device [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Driver failed to attach volume bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:57.167 2114 ERROR nova.compute.manager [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failed to attach bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     connector)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:59.292 2114 ERROR oslo.messaging.rpc.dispatcher [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Exception during message handling: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 414, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     payload)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     pass
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 284, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4549, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4547, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     bdm.destroy(context)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4544, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4567, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, bdm.volume_id)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     connector)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     mountpoint)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     instance_name, dev_number, hotplug)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return session.xenapi_request(method, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher
  
  Nova config file: /etc/nova/nova.conf
  
  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone
  
  network_api_class = nova.network.api.API
  security_group_api = nova
  network_manager = nova.network.manager.FlatDHCPManager
  allow_same_net_traffic = True
  multi_host = True
  send_arp_for_ha = True
  share_dhcp_address = True
  force_dhcp_release = True
  flat_network_bridge = xenbr0
  flat_interface = eth1
  public_interface = eth0
  volume_api_class=nova.volume.cinder.API
  
  firewall_driver = nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
  volume_api_class=nova.volume.cinder.API
  debug = True
  verbose = True
  
- 
  qpid_hostname=192.168.1.7
  qpid_port=5672
  
  Cinder config file: /etc/cinder/cinder.conf
- 
  
  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone
  my_ip = 192.168.1.107
  glance_host = 192.168.1.7
  iscsi_helper = tgtadm
  verbose = True
  
  enabled_backends=lvm
  
- 
  volume.log
  
  2015-01-07 15:42:04.726 INFO cinder.brick.iscsi.iscsi [req-1b493d4d-18d7-4be9-9c23-0edf17a2c9ce 4389a9a7146844c0806ed9b756c69b0d e303a4d222b04e9f9217d1366eb8d945 - - -] Creating iscsi_target for: volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
  2015-01-07 15:42:04.794 2162 INFO cinder.volume.manager [-] Updating volume status
  2015-01-07 15:42:05.048 2162 INFO cinder.volume.manager [-] Updating volume replication status.

** Description changed:

- I am not able to attach Cinder volumes to instances inside XenServer. My
- storage node is a PV guest running inside XenServer with 200 GB cinder-
- volumes VG, and with tgt service configured as follow:
+ I am not able to attach Cinder Juno volumes to instances inside
+ XenServer. My storage node is a PV guest running inside XenServer with
+ 200 GB cinder-volumes VG, and with tgt service configured as follow:
  
  /etc/tgt/targets.conf
  
  include /etc/tgt/conf.d/cinder.conf
  # Set the driver. If not specified, defaults to "iscsi".
  default-driver iscsi
  
  /etc/tgt/conf.d/cinder.conf
  
  include /etc/cinder/volumes/*
  
  The volume is created successfully as I can see it from Horizon
  dashboard, and also the LUN from tgtadm output:
  
  tgtadm --op show --mode target
  
  Target 2: iqn.2010-10.org.openstack:volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
      System information:
          Driver: iscsi
          State: ready
      I_T nexus information:
      LUN information:
          LUN: 0
              Type: controller
              SCSI ID: IET     00020000
              SCSI SN: beaf20
              Size: 0 MB, Block size: 1
              Online: Yes
              Removable media: No
              Prevent removal: No
              Readonly: No
              SWP: No
              Thin-provisioning: No
              Backing store type: null
              Backing store path: None
              Backing store flags:
          LUN: 1
              Type: disk
              SCSI ID: IET     00020001
              SCSI SN: beaf21
              Size: 1074 MB, Block size: 512
              Online: Yes
              Removable media: No
              Prevent removal: No
              Readonly: No
              SWP: No
              Thin-provisioning: No
              Backing store type: rdwr
              Backing store path: /dev/cinder-volumes/volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
              Backing store flags:
      Account information:
          a4ym7KcCR7TZ6sLj8sgm
      ACL information:
          ALL
  
  When I try to attach the volume to an instance, the following error is
  thrown on compute node:
  
  nova-manage.log
  
  WARNING nova.volume.cinder [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
  2015-01-07 15:42:56.464 2114 WARNING nova.virt.xenapi.volume_utils [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Ignoring exception ['VDI_IN_USE', 'OpaqueRef:2225b8fb-0a06-f859-0648-09f71aee330e'] when unplugging PBD OpaqueRef:54856f0b-bcca-a114-c96c-a86a528d3761
  2015-01-07 15:42:56.507 2114 ERROR root [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 67, in _attach_volume\n    dev_number, hotplug)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 124, in _attach_volume_to_vm\n    self._session.VBD.plug(vbd_ref, vm_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 84, in plug\n    synchronized_plug()\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 79, in synchronized_plug\n    self._call_method("plug", vbd_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 59, in _call_method\n    return self.session.call_xenapi(call, *args)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi\n    return session.xenapi_request(method, args)\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request\n    result = _parse_result(getattr(self, methodname)(*full_params))\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result\n    raise Failure(result[\'ErrorDescription\'])\n', "Failure: ['SR_BACKEND_FAILURE_46', '', 'The VDI is not available', '']\n"]
  2015-01-07 15:42:56.508 2114 ERROR nova.virt.block_device [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Driver failed to attach volume bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:57.167 2114 ERROR nova.compute.manager [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failed to attach bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     connector)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:59.292 2114 ERROR oslo.messaging.rpc.dispatcher [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Exception during message handling: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 414, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     payload)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     pass
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 284, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4549, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4547, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     bdm.destroy(context)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4544, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4567, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, bdm.volume_id)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     connector)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     mountpoint)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     instance_name, dev_number, hotplug)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return session.xenapi_request(method, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher
  
  Nova config file: /etc/nova/nova.conf
  
  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone
  
  network_api_class = nova.network.api.API
  security_group_api = nova
  network_manager = nova.network.manager.FlatDHCPManager
  allow_same_net_traffic = True
  multi_host = True
  send_arp_for_ha = True
  share_dhcp_address = True
  force_dhcp_release = True
  flat_network_bridge = xenbr0
  flat_interface = eth1
  public_interface = eth0
  volume_api_class=nova.volume.cinder.API
  
  firewall_driver = nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
  volume_api_class=nova.volume.cinder.API
  debug = True
  verbose = True
  
  qpid_hostname=192.168.1.7
  qpid_port=5672
  
  Cinder config file: /etc/cinder/cinder.conf
  
  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone
  my_ip = 192.168.1.107
  glance_host = 192.168.1.7
  iscsi_helper = tgtadm
  verbose = True
  
  enabled_backends=lvm
  
  volume.log
  
  2015-01-07 15:42:04.726 INFO cinder.brick.iscsi.iscsi [req-1b493d4d-18d7-4be9-9c23-0edf17a2c9ce 4389a9a7146844c0806ed9b756c69b0d e303a4d222b04e9f9217d1366eb8d945 - - -] Creating iscsi_target for: volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
  2015-01-07 15:42:04.794 2162 INFO cinder.volume.manager [-] Updating volume status
  2015-01-07 15:42:05.048 2162 INFO cinder.volume.manager [-] Updating volume replication status.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1408329

Title:
  Error attaching volumes to instances on XenServer

Status in OpenStack Compute (Nova):
  New

Bug description:
  I am not able to attach Cinder Juno volumes to instances inside
  XenServer. My storage node is a PV guest running inside XenServer with
  200 GB cinder-volumes VG, and with tgt service configured as follow:

  /etc/tgt/targets.conf

  include /etc/tgt/conf.d/cinder.conf
  # Set the driver. If not specified, defaults to "iscsi".
  default-driver iscsi

  /etc/tgt/conf.d/cinder.conf

  include /etc/cinder/volumes/*

  The volume is created successfully as I can see it from Horizon
  dashboard, and also the LUN from tgtadm output:

  tgtadm --op show --mode target

  Target 2: iqn.2010-10.org.openstack:volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
      System information:
          Driver: iscsi
          State: ready
      I_T nexus information:
      LUN information:
          LUN: 0
              Type: controller
              SCSI ID: IET     00020000
              SCSI SN: beaf20
              Size: 0 MB, Block size: 1
              Online: Yes
              Removable media: No
              Prevent removal: No
              Readonly: No
              SWP: No
              Thin-provisioning: No
              Backing store type: null
              Backing store path: None
              Backing store flags:
          LUN: 1
              Type: disk
              SCSI ID: IET     00020001
              SCSI SN: beaf21
              Size: 1074 MB, Block size: 512
              Online: Yes
              Removable media: No
              Prevent removal: No
              Readonly: No
              SWP: No
              Thin-provisioning: No
              Backing store type: rdwr
              Backing store path: /dev/cinder-volumes/volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
              Backing store flags:
      Account information:
          a4ym7KcCR7TZ6sLj8sgm
      ACL information:
          ALL

  When I try to attach the volume to an instance, the following error is
  thrown on compute node:

  nova-manage.log

  WARNING nova.volume.cinder [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
  2015-01-07 15:42:56.464 2114 WARNING nova.virt.xenapi.volume_utils [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] Ignoring exception ['VDI_IN_USE', 'OpaqueRef:2225b8fb-0a06-f859-0648-09f71aee330e'] when unplugging PBD OpaqueRef:54856f0b-bcca-a114-c96c-a86a528d3761
  2015-01-07 15:42:56.507 2114 ERROR root [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 67, in _attach_volume\n    dev_number, hotplug)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 124, in _attach_volume_to_vm\n    self._session.VBD.plug(vbd_ref, vm_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 84, in plug\n    synchronized_plug()\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 79, in synchronized_plug\n    self._call_method("plug", vbd_ref)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/objects.py", line 59, in _call_method\n    return self.session.call_xenapi(call, *args)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi\n    return session.xenapi_request(method, args)\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request\n    result = _parse_result(getattr(self, methodname)(*full_params))\n', '  File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result\n    raise Failure(result[\'ErrorDescription\'])\n', "Failure: ['SR_BACKEND_FAILURE_46', '', 'The VDI is not available', '']\n"]
  2015-01-07 15:42:56.508 2114 ERROR nova.virt.block_device [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Driver failed to attach volume bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:56.508 2114 TRACE nova.virt.block_device [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:57.167 2114 ERROR nova.compute.manager [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 None] [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failed to attach bdfb5407-bea9-42d8-8baa-a3dfd0547fe9 at /dev/xvdd
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Traceback (most recent call last):
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     connector)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     mountpoint)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     instance_name, dev_number, hotplug)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     return session.xenapi_request(method, args)
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6] Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:57.167 2114 TRACE nova.compute.manager [instance: 1fe0b7f2-ddeb-4edc-9375-12097502ecd6]
  2015-01-07 15:42:59.292 2114 ERROR oslo.messaging.rpc.dispatcher [req-8cc840c2-1192-4d8a-a71f-d028e6552f29 ] Exception during message handling: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 414, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     payload)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     pass
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 284, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 326, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4549, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4547, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     bdm.destroy(context)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4544, in do_attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return self._attach_volume(context, instance, driver_bdm)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4567, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, bdm.volume_id)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4559, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 261, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     connector)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 252, in attach
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 416, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     mountpoint)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 45, in attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     instance_name, dev_number, hotplug)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volumeops.py", line 75, in _attach_volume
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     volume_utils.forget_sr(self._session, sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/volume_utils.py", line 240, in forget_sr
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     session.call_xenapi("SR.forget", sr_ref)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/client/session.py", line 187, in call_xenapi
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     return session.xenapi_request(method, args)
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 133, in xenapi_request
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     result = _parse_result(getattr(self, methodname)(*full_params))
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/XenAPI.py", line 203, in _parse_result
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher     raise Failure(result['ErrorDescription'])
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher Failure: ['SR_HAS_PBD', 'OpaqueRef:ac31db07-fafd-e15e-fe70-3ae7bc64b26c']
  2015-01-07 15:42:59.292 2114 TRACE oslo.messaging.rpc.dispatcher

  Nova config file: /etc/nova/nova.conf

  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone

  network_api_class = nova.network.api.API
  security_group_api = nova
  network_manager = nova.network.manager.FlatDHCPManager
  allow_same_net_traffic = True
  multi_host = True
  send_arp_for_ha = True
  share_dhcp_address = True
  force_dhcp_release = True
  flat_network_bridge = xenbr0
  flat_interface = eth1
  public_interface = eth0
  volume_api_class=nova.volume.cinder.API

  firewall_driver = nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
  volume_api_class=nova.volume.cinder.API
  debug = True
  verbose = True

  qpid_hostname=192.168.1.7
  qpid_port=5672

  Cinder config file: /etc/cinder/cinder.conf

  [DEFAULT]
  rpc_backend = qpid
  auth_strategy = keystone
  my_ip = 192.168.1.107
  glance_host = 192.168.1.7
  iscsi_helper = tgtadm
  verbose = True

  enabled_backends=lvm

  volume.log

  2015-01-07 15:42:04.726 INFO cinder.brick.iscsi.iscsi [req-1b493d4d-18d7-4be9-9c23-0edf17a2c9ce 4389a9a7146844c0806ed9b756c69b0d e303a4d222b04e9f9217d1366eb8d945 - - -] Creating iscsi_target for: volume-bdfb5407-bea9-42d8-8baa-a3dfd0547fe9
  2015-01-07 15:42:04.794 2162 INFO cinder.volume.manager [-] Updating volume status
  2015-01-07 15:42:05.048 2162 INFO cinder.volume.manager [-] Updating volume replication status.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1408329/+subscriptions
Follow ups

[Bug 1408329] Re: Error attaching volumes to instances on XenServer
From: nedo, 2015-01-11
[Bug 1408329] [NEW] Error attaching volumes to instances on XenServer
From: nedo, 2015-01-07
References

[Bug 1408329] [NEW] Error attaching volumes to instances on XenServer
From: nedo, 2015-01-07