yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1727369] [NEW] _supports_direct_io() check fails on shared storage

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pavel Gluschak <trustytwelve@xxxxxxxxx>
Date: Wed, 25 Oct 2017 14:31:09 -0000
Reply-to: Bug 1727369 <1727369@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When instances are deployed on multiple compute nodes concurrently and
instance_path is set to shared storage (i.e. Virtuozzo Storage), that
doesn't support concurrent write operations on the same file,
_supports_direct_io() fails with OSError: [Errno 16] Device or resource
busy: '/var/lib/nova/instances/.directio.test'.

2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] Traceback (most recent call last):
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     yield resources
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     block_device_info=block_device_info)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2941, in spawn
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     write_to_disk=True)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4909, in _get_guest_xml
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     context)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4768, in _get_guest_config
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     flavor, guest.os_type)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3806, in _get_guest_storage_config
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     inst_type)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3748, in _get_guest_disk_config
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.disk_cachemode,
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 739, in disk_cachemode
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     if not self._supports_direct_io(CONF.instances_path):
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3128, in _supports_direct_io
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     {'path': dirpath, 'ex': e})
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.force_reraise()
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     six.reraise(self.type_, self.value, self.tb)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3110, in _supports_direct_io
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/.directio.test'

** Affects: nova
     Importance: Undecided
     Assignee: Pavel Gluschak (scsnow)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Pavel Gluschak (scsnow)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1727369

Title:
  _supports_direct_io() check fails on shared storage

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  When instances are deployed on multiple compute nodes concurrently and
  instance_path is set to shared storage (i.e. Virtuozzo Storage), that
  doesn't support concurrent write operations on the same file,
  _supports_direct_io() fails with OSError: [Errno 16] Device or
  resource busy: '/var/lib/nova/instances/.directio.test'.

  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] Traceback (most recent call last):
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     yield resources
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     block_device_info=block_device_info)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2941, in spawn
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     write_to_disk=True)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4909, in _get_guest_xml
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     context)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4768, in _get_guest_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     flavor, guest.os_type)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3806, in _get_guest_storage_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     inst_type)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3748, in _get_guest_disk_config
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.disk_cachemode,
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 739, in disk_cachemode
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     if not self._supports_direct_io(CONF.instances_path):
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3128, in _supports_direct_io
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     {'path': dirpath, 'ex': e})
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     self.force_reraise()
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     six.reraise(self.type_, self.value, self.tb)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3110, in _supports_direct_io
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0]     f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
  2017-10-19 21:11:29.030 160580 ERROR nova.compute.manager [instance: 8ce8f6f4-a3a5-457e-bd18-72b453efe2e0] OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/.directio.test'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1727369/+subscriptions
Follow ups

[Bug 1727369] Re: _supports_direct_io() check fails on shared storage
From: OpenStack Infra, 2018-12-08