yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1632173@xxxxxxxxxxxxxxxxxx>]

[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1632173

Title:
  migrate_flavor_data doesn't flavor migrate meta data of VMs spawned
  during upgrade

Status in OpenStack Compute (nova):
  Expired

Bug description:
  Hi,

  I am upgrading from Juno to Kilo and from that to Liberty.

  I understand I need to nova-manage db migrate_flavor_data before
  upgrading from Kilo to Liberty to let VMs that were spawned while the
  system was in Juno to flavor migrate to Kilo.

  Depending on the number of computes, complete upgrade can potentially
  be spanned for longer duration, days if not months.

  While migrate_flavor_data seem to flavor migrate meta data of the VMs
  that were spawned before upgrade procedure, it doesn't seem to flavor
  migrate for the VMs that were spawned during the upgrade procedure
  more specifically after openstack controller upgrade and before
  compute upgrade. Am I missing something here or is it by intention?

  Since, the compute upgrade procedure could last for days, would it be
  practical to block spawning work load VMs for that long duration?
  Otherwise, next upgrade will fail right?

  thanks

  >> While migrate_flavor_data seem to flavor migrate meta data of the VMs
  >> that were spawned before upgrade procedure, it doesn't seem to flavor
  >> migrate for the VMs that were spawned during the upgrade procedure more
  >> specifically after openstack controller upgrade and before compute
  >> upgrade. Am I missing something here or is it by intention?

  >You can run the flavor migration as often as you need, and can certainly
  >run it after your last compute is upgraded before you start to move into
  >liberty.
  >
  >--Dan

  
  Thanks Dan for your response. While I do run that before I start my move to liberty, what I see is that it doesn't seem to flavor migrate meta data for the VMs that are spawned after controller upgrade from juno to kilo and before all computes upgraded from juno to kilo. The current work around is to delete those VMs that are spawned after controller upgrade and before all computes upgrade, and then initiate liberty upgrade. Then it works fine.

  Dan Smith <dms@xxxxxxxxxxxxx>
  	
  Aug 31
  	
  to me, openstack-dev
  > Thanks Dan for your response. While I do run that before I start my
  > move to liberty, what I see is that it doesn't seem to flavor migrate
  > meta data for the VMs that are spawned after controller upgrade from
  > juno to kilo and before all computes upgraded from juno to kilo. The
  > current work around is to delete those VMs that are spawned after
  > controller upgrade and before all computes upgrade, and then initiate
  > liberty upgrade. Then it works fine.

  I can't think of any reason why that would be, or why it would be a
  problem. Instances created after the controllers are upgraded should not
  have old-style flavor info, so they need not be touched by the migration
  code.

  Maybe filing a bug is in order describing what you see?

  
  Hi,

  I did some investigation last week, why this was happening before I
  file the bug. Here are my observations and I would like to work on it.
  Any guidance is appreciated.

  
  Upgrade procedure:
  In my setup I have openstack controller on one node, neutron on another node and rest of the nodes are computes. Assume they are running juno. As the first step,
  a) Bringup up a new openstack node in kilo
  b) Migrate database from juno to kilo.
  c) Run xxx manage db sync commands.
  d) Set upgrade levels to juno
  e) Migrate neutron and computes to the newer controller.

  second step.
  a) Upgrade neutron, and then start compute upgrades one at a time.

  third step,
  a) Finalize the upgrade by clearing upgrade levels and restarting services.
  b) Issue migrate flavor data command.

  This procedure is openstack side by side upgraded and neutron and
  computes inplace upgraded.

  Issue:
  VMs spawned after step1 and during step2 are not flavor migrated. So, migration to liberty was failing.

  Investigation:
  flavor migrate command filters instance uuids in nova.instance_extra table whose 'flavor' column is NULL and fill in with the necessary flavor information and delete those respective rows from the nova.instance_system_metadata associated with that VM.
  For the VMs spawned after the first step and before step2, this flavor information in nova.instance_extra and also respective flavor information in nova.instance_system_metadata table.
  Since the filter looks for 'flavor ' column as NULL, these instances are not caught to delete entries in nova.instance_system_metadata. And presence of this data is failing nova db sync while migrating to liberty.

  Possible Solutions:
  I feel we should broaden the filter so that instances spawned after step1 and during step 2 are also accounted for data translation. As a quick verification, I tried this by having no filer and it worked. I know filter give efficiency, but would it matter in this context or should we broaden the filter?

  Please let me know if I am going in the right direction.

  thanks

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1632173/+subscriptions