yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Changed in: nova/pike
       Status: New => Confirmed

** Changed in: nova/pike
   Importance: Undecided => Medium

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/queens
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784074

Title:
  Instances end up with no cell assigned in instance_mappings

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) pike series:
  Confirmed
Status in OpenStack Compute (nova) queens series:
  Confirmed

Bug description:
  There has been situations where due to an unrelated issue such as an
  RPC or DB problem, the nova_api instance_mappings table can end up
  with instances that have cell_id set to NULL which can cause annoying
  and weird behaviour such as undeletable instances, etc.

  This seems to be an issue only during times where these external
  infrastructure components had issues.  I have come up with the
  following script which loops over all cells and checks where they are,
  and spits out a mysql query to run to fix.

  This would be nice to have as a nova-manage cell_v2 command to help if
  any other users run into this, unfortunately I'm a bit short on time
  so I don't have time to nova-ify it, but it's here:

  ========================================================================
  #!/usr/bin/env python

  import urlparse

  import pymysql

  
  # Connect to databases
  api_conn = pymysql.connect(host='xxxx', port=3306, user='nova_api', passwd='xxxxxxx', db='nova_api')
  api_cur = api_conn.cursor()

  def _get_conn(db):
    parsed_url = urlparse.urlparse(db)
    conn = pymysql.connect(host=parsed_url.hostname, user=parsed_url.username, passwd=parsed_url.password, db=parsed_url.path[1:])
    return conn.cursor()

  # Get list of all cells
  api_cur.execute("SELECT uuid, name, database_connection FROM cell_mappings")
  CELLS = [{'uuid': uuid, 'name': name, 'db': _get_conn(db)} for uuid, name, db in api_cur.fetchall()]

  # Get list of all unmapped instances
  api_cur.execute("SELECT instance_uuid FROM instance_mappings WHERE cell_id IS NULL")
  print "Number of unmapped instances: %s" % api_cur.rowcount

  # Go over all unmapped instances
  for (instance_uuid,) in api_cur.fetchall():
    instance_cell = None

    # Check which cell contains this instance
    for cell in CELLS:
      cell['db'].execute("SELECT id FROM instances WHERE uuid = %s", (instance_uuid,))

      if cell['db'].rowcount != 0:
        instance_cell = cell
        break

    # Update to the correct cell
    if instance_cell:
      print "UPDATE instance_mappings SET cell_id = '%s' WHERE instance_uuid = '%s'" % (instance_cell['uuid'], instance_uuid)
      continue

    # If we reach this point, it's not in any cell?!
    print "%s: not found in any cell" % (instance_uuid)
  ========================================================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1784074/+subscriptions