yahoo-eng-team team mailing list archive

[OpenStack Infra <1727369@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/515091
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=26521718bdba3bccbf6270e26b76754c26304658
Submitter: Zuul
Branch:    master

commit 26521718bdba3bccbf6270e26b76754c26304658
Author: Pavel Glushchak <pglushchak@xxxxxxxxxxxxx>
Date:   Wed Oct 25 15:58:01 2017 +0300

    Fixed concurrent access to direct io test file
    
    When instances are deployed concurrently on multiple
    compute nodes, nova-compute may fail on checking
    direct io capabilities when instance_path is shared
    across nodes. This fails on shared storage, that
    doesn't support concurrent write access to the
    same file, i.e. Virtuozzo Storage. To fix that we add
    random string at the end of test file name.
    
    Closes-Bug: #1727369
    Change-Id: I085ade355fba0e9727a38d2dcbc9cffa735a62d1
    Signed-off-by: Pavel Glushchak <pglushchak@xxxxxxxxxxxxx>


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1727369

Title:
  _supports_direct_io() check fails on shared storage

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  When instances are deployed on multiple compute nodes concurrently and
  instance_path is set to shared storage (i.e. Virtuozzo Storage), that
  doesn't support concurrent write operations on the same file,
  _supports_direct_io() may fail with OSError: [Errno 16] Device or
  resource busy: '/var/lib/nova/instances/.directio.test'.

  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] Traceback (most recent call last):
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     yield resources
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     block_device_info=block_device_info)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2941, in spawn
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     write_to_disk=True)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4909, in _get_guest_xml
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     context)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4768, in _get_guest_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     flavor, guest.os_type)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3806, in _get_guest_storage_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     inst_type)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3748, in _get_guest_disk_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.disk_cachemode,
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 739, in disk_cachemode
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     if not self._supports_direct_io(CONF.instances_path):
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3128, in _supports_direct_io
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     {'path': dirpath, 'ex': e})
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.force_reraise()
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     six.reraise(self.type_, self.value, self.tb)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3110, in _supports_direct_io
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/.directio.test'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1727369/+subscriptions