← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #63623

[Bug 1502028] Re: cannot attach a volume when using multiple ceph backends

  Thread Previous • Date Previous • • Thread Next 
As mentioned in comment 8, I think this got fixed by
https://review.openstack.org/#/c/389399 in Ocata (15.0.0.0b2).

As for a backport, the best way is to ask in #openstack-nova in IRC or
in a Nova meeting or on the openstack-dev mailing list.

** Changed in: nova
       Status: Confirmed => Fix Released

** Changed in: nova
     Assignee: Kevin Zhao (kevin-zhao) => Matt Riedemann (mriedem)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1502028

Title:
  cannot attach a volume when using multiple ceph backends

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  1. Exact version of Nova/OpenStack you are running: Kilo Stable

  2. Relevant log files:

  I'm testing using ceph RADOS block devices to attach VM; however I've
  hit an issue when ceph cluster is different between VM and volumes.

  <--error message-->
  2015-09-24 11:32:31 13083 DEBUG nova.virt.libvirt.config [req-b9bbd744-cf75-477b-b6a6-ea5b72f6181f 9504f2c4fe6b4b34a1bb0330f2faba35 0788824d5d1f46f2b014597ba8dc0585] Generated XML ('<disk type="network" device="disk">\n  <driver name="qemu" type="raw" cache="none"/>\n  <source protocol="rbd" name="rbd/volume-727c5319-1926-44ac-ba52-de55485faf2b">\n    <host name="10.40.100.115" port="6789"/>\n    <host name="10.40.100.116" port="6789"/>\n    <host name="10.40.100.119" port="6789"/>\n  </source>\n  <auth username="cinder">\n    <secret type="ceph" uuid="457eb676-33da-42ec-9a8c-9293d545c337"/>\n  </auth>\n  <target bus="virtio" dev="vdb"/>\n  <serial>727c5319-1926-44ac-ba52-de55485faf2b</serial>\n</disk>\n',)  to_xml /opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/nova/virt/libvirt/config.py:82
  2015-09-24 11:32:31 13083 ERROR nova.virt.libvirt.driver [req-b9bbd744-cf75-477b-b6a6-ea5b72f6181f 9504f2c4fe6b4b34a1bb0330f2faba35 0788824d5d1f46f2b014597ba8dc0585] [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] Failed to attach volume at mountpoint: /dev/vdb
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] Traceback (most recent call last):
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     virt_dom.attachDeviceFlags(conf.to_xml(), flags)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     rv = execute(f, *args, **kwargs)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     six.reraise(c, e, tb)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     rv = meth(*args, **kwargs)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/libvirt.py", line 528, in attachDeviceFlags
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2015-09-24 11:32:31.923 13083 TRACE nova.virt.libvirt.driver [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]
  2015-09-24 11:32:31 13083 ERROR nova.virt.block_device [req-b9bbd744-cf75-477b-b6a6-ea5b72f6181f 9504f2c4fe6b4b34a1bb0330f2faba35 0788824d5d1f46f2b014597ba8dc0585] [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] Driver failed to attach volume 727c5319-1926-44ac-ba52-de55485faf2b at /dev/vdb
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] Traceback (most recent call last):
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/nova/virt/block_device.py", line 255, in attach
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     device_type=self['device_type'], encryption=encryption)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1103, in attach_volume
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     self._disconnect_volume(connection_info, disk_dev)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     six.reraise(self.type_, self.value, self.tb)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     virt_dom.attachDeviceFlags(conf.to_xml(), flags)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     rv = execute(f, *args, **kwargs)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     six.reraise(c, e, tb)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     rv = meth(*args, **kwargs)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]   File "/opt/stack/venv/nova-20150831T151915Z/lib/python2.7/site-packages/libvirt.py", line 528, in attachDeviceFlags
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2015-09-24 11:32:31.926 13083 TRACE nova.virt.block_device [instance: 3aa05494-88ef-44c3-a7ad-705437b5f165]

  This is a VM libvirt xml. A root disk is created in "ceph1" cluster.

  <--xml-->
      <disk type='network' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <auth username='cinder'>
          <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
        </auth>
        <source protocol='rbd' name='volumes/def50421-13d9-4bbd-ad93-4f95b9d38bf6_disk'>
          <host name='10.40.100.36' port='6789'/>
          <host name='10.40.100.37' port='6789'/>
          <host name='10.40.100.38' port='6789'/>
        </source>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </disk>

  
  but in my environment, I have two ceph clusters and use multi backends and volume is created in "ssdceph" cluster. 

  <--cinder config for multi backends-->
  # Configure the enabled backends
  enabled_backends= ceph1,ssdceph

  [ssdceph]
  rbd_max_clone_depth = 5
  rbd_flatten_volume_from_snapshot = False
  rbd_user = admin
  rbd_pool = rbd
  rbd_ceph_conf = /etc/ceph/ssd/ceph.conf
  volume_driver = cinder.volume.drivers.rbd.RBDDriver
  volume_backend_name = ssdceph
  rbd_secret_uuid = 7b16f0cb-1276-4ba5-a7c5-74e2bb06d836

  [ceph1]
  rbd_max_clone_depth = 5
  rbd_flatten_volume_from_snapshot = False
  rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
  rbd_user = cinder
  rbd_pool = volumes
  rbd_ceph_conf = /etc/ceph/ceph.conf

  backup_driver = cinder.backup.drivers.ceph
  backup_ceph_conf = /etc/ceph/ceph.conf
  backup_ceph_user = cinder-backup
  backup_ceph_chunk_size = 134217728
  backup_ceph_pool = backups
  backup_ceph_stripe_unit = 0
  backup_ceph_stripe_count = 0
  restore_discard_excess_bytes = true
  volume_driver = cinder.volume.drivers.rbd.RBDDriver
  volume_backend_name = ceph

  3. Reproduce steps:
  (1) Create VM "X" with "A" ceph cluster. A root disk is created in "A" ceph cluster.
  (2) Create volume "Y" with "B" ceph cluster using multi backends in cinder.
  (3) Attach volume "Y" to "X", libvirt can't attach a volume properly.

  Expected result: volume status is "in-use" and volume attached.
  Actual result: volume status back to "available" and libvirt can't attach a volume properly.

  nova-compute try to attach a volume using the secret_uuid, username
  parameter in nova.conf. If a volume has different secret_uuid,
  username with nova.conf, libvirt can't find exact volume in ceph
  cluster and error occured.

  If a backend of volume which try to attach is ceph, user and
  secret_uuid information of volume should be provided by cinder; not in
  nova.conf.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1502028/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next