yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1845199] [NEW] Nova-compute doesn't mount volume filesystems on resume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jacek Tomasiak <1845199@xxxxxxxxxxxxxxxxxx>
Date: Tue, 24 Sep 2019 11:33:23 -0000
Reply-to: Bug 1845199 <1845199@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
Suspended instances with volumes attached fail to resume if the /var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due to e.g. node reboot during upgrade.

Steps to reproduce
==================
* Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/... should get mounted on the compute node)
* Suspend the instance ($ nova suspend <instance id or name>)
* Unmount the /var/lib/nova/mnt/... filesystem
* Try to resume the instance ($ nova resume <instance id or name>)

Expected result
===============
The instance should become ACTIVE.

Actual result
=============
Instance switches to ERROR state.

Environment
===========
openstack-nova-compute-18.2.3~dev10-3.5.noarch
python-novaclient-11.0.0-3.126.noarch
python-nova-18.2.3~dev10-3.5.noarch
openstack-nova-18.2.3~dev10-3.5.noarch

Hypervisor: Libvirt + KVM

Storage: We experienced this with NFS Cinder backend but probably other
filesystem-based volume types are affected as well.

Network: Neutron

Logs & Configs
==============
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2 3dab9b31bfaf4298801706aaf14ce83a - default default] [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR: libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in _error_out_instance_on_exception
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     yield
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in resume_instance
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     block_device_info)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in resume
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     vifs_already_plugged=True)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     destroy_disks_on_failure)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     post_xml_callback=post_xml_callback)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in _create_domain
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     guest.launch(pause=pause)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self._encoded_xml, errors='ignore')
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     return self._domain.createWithFlags(flags)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     rv = execute(f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(c, e, tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     rv = meth(*args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1845199

Title:
  Nova-compute doesn't mount volume filesystems on resume

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Suspended instances with volumes attached fail to resume if the /var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due to e.g. node reboot during upgrade.

  Steps to reproduce
  ==================
  * Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/... should get mounted on the compute node)
  * Suspend the instance ($ nova suspend <instance id or name>)
  * Unmount the /var/lib/nova/mnt/... filesystem
  * Try to resume the instance ($ nova resume <instance id or name>)

  Expected result
  ===============
  The instance should become ACTIVE.

  Actual result
  =============
  Instance switches to ERROR state.

  Environment
  ===========
  openstack-nova-compute-18.2.3~dev10-3.5.noarch
  python-novaclient-11.0.0-3.126.noarch
  python-nova-18.2.3~dev10-3.5.noarch
  openstack-nova-18.2.3~dev10-3.5.noarch

  Hypervisor: Libvirt + KVM

  Storage: We experienced this with NFS Cinder backend but probably
  other filesystem-based volume types are affected as well.

  Network: Neutron

  Logs & Configs
  ==============
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2 3dab9b31bfaf4298801706aaf14ce83a - default default] [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR: libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in _error_out_instance_on_exception
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     yield
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in resume_instance
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     block_device_info)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in resume
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     vifs_already_plugged=True)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in _create_domain_and_network
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     destroy_disks_on_failure)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self.force_reraise()
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(self.type_, self.value, self.tb)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in _create_domain_and_network
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     post_xml_callback=post_xml_callback)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in _create_domain
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     guest.launch(pause=pause)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self._encoded_xml, errors='ignore')
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     self.force_reraise()
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(self.type_, self.value, self.tb)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     return self._domain.createWithFlags(flags)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     rv = execute(f, *args, **kwargs)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     six.reraise(c, e, tb)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     rv = meth(*args, **kwargs)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1845199/+subscriptions