yahoo-eng-team team mailing list archive

[Bjoern Teipel <bjoern.teipel@xxxxxxxxxxxxx>]

Public bug reported:

Description
===========

Instance flavors with Ephemeral/Swap disk configuration created under
Juno do not live migrate/migrate/resize cleanly due to a change in
Liberty as the naming scheme for these underlying disk have changed
around the file extension from

ephemeral_size_fstype

Example: /var/lib/nova/instances/_base/ephemeral_100_default

to

ephemeral_size_hash

Example: /var/lib/nova/instances/_base/ephemeral_100_40d1d2c

per


https://github.com/openstack/nova/blob/juno-eol/nova/virt/disk/api.py#L102

def get_fs_type_for_os_type(os_type):
    return os_type if _MKFS_COMMAND.get(os_type) else 'default'

https://github.com/openstack/nova/blob/liberty-
eol/nova/virt/disk/api.py#L123

def get_file_extension_for_os_type(os_type, specified_fs=None):
    mkfs_command = _MKFS_COMMAND.get(os_type, _DEFAULT_MKFS_COMMAND)
    if mkfs_command:
        extension = mkfs_command
    else:
        if not specified_fs:
            specified_fs = CONF.default_ephemeral_format
            if not specified_fs:
                specified_fs = _DEFAULT_FS_BY_OSTYPE.get(os_type,
                                                         _DEFAULT_FILE_SYSTEM)
        extension = specified_fs
    return utils.get_hash_str(extension)[:7]

which are used the create the file extension.

This leads to errors like

2018-12-15 23:19:40.212+0000: 16105: error :
virStorageFileGetMetadataRecurse:3063 : Cannot access backing file
'/var/lib/nova/instances/_base/swap_2048' of storage file
'/var/lib/nova/instances/a2202625-c9a4-4855-acde-de539d8059ef/disk.swap'
(as uid:108, gid:115): No such file or directory

during migration and you're forced to perform a offline migration to rebuild and reformat
the ephemeral storage


Steps to reproduce
==================

- Create instance with Juno using a flavor which uses ephemeral/swap disks
- Upgrade OpenStack to Newton (because migration was too buggy prior)
- Attempt migration


Expected result
===============

Migration succeeds and honors the file extension format


Actual result
=============
Migrations fail and instances go into error state


Environment
===========

Errors were encountered with newton-eol code but
looking at the code it would happen with nova code past Juno, including all current releases.


Logs & Configs
==============

N/A

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1813188

Title:
  Live migrations break on flavors with swap/ephemeral disks when
  created on Juno

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  Instance flavors with Ephemeral/Swap disk configuration created under
  Juno do not live migrate/migrate/resize cleanly due to a change in
  Liberty as the naming scheme for these underlying disk have changed
  around the file extension from

  ephemeral_size_fstype

  Example: /var/lib/nova/instances/_base/ephemeral_100_default

  to

  ephemeral_size_hash

  Example: /var/lib/nova/instances/_base/ephemeral_100_40d1d2c

  per

  
  https://github.com/openstack/nova/blob/juno-eol/nova/virt/disk/api.py#L102

  def get_fs_type_for_os_type(os_type):
      return os_type if _MKFS_COMMAND.get(os_type) else 'default'

  https://github.com/openstack/nova/blob/liberty-
  eol/nova/virt/disk/api.py#L123

  def get_file_extension_for_os_type(os_type, specified_fs=None):
      mkfs_command = _MKFS_COMMAND.get(os_type, _DEFAULT_MKFS_COMMAND)
      if mkfs_command:
          extension = mkfs_command
      else:
          if not specified_fs:
              specified_fs = CONF.default_ephemeral_format
              if not specified_fs:
                  specified_fs = _DEFAULT_FS_BY_OSTYPE.get(os_type,
                                                           _DEFAULT_FILE_SYSTEM)
          extension = specified_fs
      return utils.get_hash_str(extension)[:7]

  which are used the create the file extension.

  This leads to errors like

  2018-12-15 23:19:40.212+0000: 16105: error :
  virStorageFileGetMetadataRecurse:3063 : Cannot access backing file
  '/var/lib/nova/instances/_base/swap_2048' of storage file
  '/var/lib/nova/instances/a2202625-c9a4-4855-acde-
  de539d8059ef/disk.swap' (as uid:108, gid:115): No such file or
  directory

  during migration and you're forced to perform a offline migration to rebuild and reformat
  the ephemeral storage

  
  Steps to reproduce
  ==================

  - Create instance with Juno using a flavor which uses ephemeral/swap disks
  - Upgrade OpenStack to Newton (because migration was too buggy prior)
  - Attempt migration

  
  Expected result
  ===============

  Migration succeeds and honors the file extension format

  
  Actual result
  =============
  Migrations fail and instances go into error state


  Environment
  ===========

  Errors were encountered with newton-eol code but
  looking at the code it would happen with nova code past Juno, including all current releases.

  
  Logs & Configs
  ==============

  N/A

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1813188/+subscriptions