yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1356534] [NEW] VMs that don't have config drive fail to start when force_config_drive=Always

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mike Dorman <mdorman@xxxxxxxxxxx>
Date: Wed, 13 Aug 2014 19:11:46 -0000
Reply-to: Bug 1356534 <1356534@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When force_config_drive=Always is set, VMs that did not previously have
a config drive created for them will fail to start.

In our particular use case, we had NOT been using config drive for a
while, and then enabled it with force_config_drive=Always.  Any VMs
created before that time did not have a config drive created, and are
now failing to start because nova-compute expects all VMs to have one.

2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     **args)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     payload)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     pass
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._power_on(context, instance)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._hard_reboot(context, instance, network_info, block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     dk_size = int(os.path.getsize(path))
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return os.stat(filename).st_size
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config'
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp

The work around for now is to just create a 0-byte file (or empty iso
image) for disk.config on the VMs that didn't previously have one.

But, in our opinion, nova should not fail to boot the VM in this
situation.  I see there as three valid behaviors:

1.  Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM
2.  Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one.
3.  Recreate the config drive disk for VMs on every boot.  This has the added advantage of making the config drive dynamic.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  When force_config_drive=Always is set, VMs that did not previously have
  a config drive created for them will fail to start.
  
  In our particular use case, we had NOT been using config drive for a
  while, and then enabled it with force_config_drive=Always.  Any VMs
  created before that time did not have a config drive created, and are
  now failing to start because nova-compute expects all VMs to have one.
  
  2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     pass
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._power_on(context, instance)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._hard_reboot(context, instance, network_info, block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     dk_size = int(os.path.getsize(path))
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return os.stat(filename).st_size
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config'
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp
  
  The work around for now is to just create a 0-byte file (or empty iso
  image) for disk.config on the VMs that didn't previously have one.
  
  But, in our opinion, nova should not fail to boot the VM in this
- situation.  I see there as two valid behaviors:
+ situation.  I see there as three valid behaviors:
  
  1.  Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM
  2.  Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one.
+ 3.  Recreate the config drive disk for VMs on every boot.  This has the added advantage of making the config drive dynamic.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1356534

Title:
  VMs that don't have config drive fail to start when
  force_config_drive=Always

Status in OpenStack Compute (Nova):
  New

Bug description:
  When force_config_drive=Always is set, VMs that did not previously
  have a config drive created for them will fail to start.

  In our particular use case, we had NOT been using config drive for a
  while, and then enabled it with force_config_drive=Always.  Any VMs
  created before that time did not have a config drive created, and are
  now failing to start because nova-compute expects all VMs to have one.

  2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     pass
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._power_on(context, instance)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     self._hard_reboot(context, instance, network_info, block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     dk_size = int(os.path.getsize(path))
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp     return os.stat(filename).st_size
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config'
  2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp

  The work around for now is to just create a 0-byte file (or empty iso
  image) for disk.config on the VMs that didn't previously have one.

  But, in our opinion, nova should not fail to boot the VM in this
  situation.  I see there as three valid behaviors:

  1.  Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM
  2.  Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one.
  3.  Recreate the config drive disk for VMs on every boot.  This has the added advantage of making the config drive dynamic.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1356534/+subscriptions
Follow ups

[Bug 1356534] [NEW] VMs that don't have config drive fail to start when force_config_drive=Always
From: Mike Dorman, 2014-08-13
References

[Bug 1356534] [NEW] VMs that don't have config drive fail to start when force_config_drive=Always
From: Mike Dorman, 2014-08-13