yahoo-eng-team team mailing list archive

[OpenStack Infra <1622545@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/377933
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ceaf853894352b6d0ae12efe85ba5eb4e651e58a
Submitter: Jenkins
Branch:    master

commit ceaf853894352b6d0ae12efe85ba5eb4e651e58a
Author: Dan Smith <dansmith@xxxxxxxxxx>
Date:   Tue Sep 27 10:17:00 2016 -0700

    Archive instance-related rows when the parent instance is deleted
    
    This is something I expect has been very broken for a long time. We
    have rows in tables such as instance_extra, instance_faults, etc that
    pertain to a single instance, and thus have a foreign key on their
    instance_uuid column that points to the instance. If any of those
    records exist, an instance can not be archived out of the main
    instances table.
    
    The archive routine currently "handles" this by skipping over said
    instances, and eventually iterating over all the tables to pull out
    any records that point to that instance, thus freeing up the instance
    itself for archival. The problem is, this only happens if those extra
    records are actually marked as deleted themselves. If we fail during
    a cleanup routine and leave some of them not marked as deleted, but
    where the instance they reference *is* marked as deleted, we will
    never archive them.
    
    This patch adds another phase of the archival process for any table
    that has an "instance_uuid" column, which attempts to archive records
    that point to these deleted instances. With this, using a very large
    real world sample database, I was able to archive my way down to
    zero deleted, un-archivable instances (from north of 100k).
    
    Closes-Bug: #1622545
    Change-Id: I77255c77780f0c2b99d59a9c20adecc85335bb18


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1622545

Title:
  archive_deleted_rows isn't archiving instances

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Running "nova-manage archive_deleted_rows ..." clears out little or
  none of the deleted nova instances

  for example running the command several times

  $ nova-manage --debug db archive_deleted_rows --max_rows 100000
  --verbose

  I get
  +--------------------------+-------------------------+
  | Table                    | Number of Rows Archived |
  +--------------------------+-------------------------+
  | block_device_mapping     | 10108                   |
  | instance_actions         | 31838                   |
  | instance_actions_events  | 2                       |
  | instance_extra           | 10108                   |
  | instance_faults          | 459                     |
  | instance_info_caches     | 10108                   |
  | instance_metadata        | 6037                    |
  | instance_system_metadata | 17883                   |
  | reservations             | 9                       |
  +--------------------------+-------------------------+

  the only way I've been able to get an instances archived is to lower
  the --max-rows parameter, but this only deletes a small number of the
  instances and sometimes doesn't archive any at all

  In my nova-mange.log I have the following error

  2016-09-12 09:22:21.658 17603 WARNING nova.db.sqlalchemy.api [-]
  IntegrityError detected when archiving table instances:
  (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent
  row: a foreign key constraint fails (`nova`.`instance_extra`,
  CONSTRAINT `instance_extra_instance_uuid_fkey` FOREIGN KEY
  (`instance_uuid`) REFERENCES `instances` (`uuid`))') [SQL: u'DELETE
  FROM instances WHERE instances.id in (SELECT T1.id FROM (SELECT
  instances.id \nFROM instances \nWHERE instances.deleted != %s ORDER BY
  instances.id \n LIMIT %s) as T1)'] [parameters: (0, 787)]

  
  mysql -e 'select count(*) from instances where deleted_at is not NULL;' nova
  +----------+
  | count(*) |
  +----------+
  |    70829 |
  +----------+

  I'm running mitaka with this patch installed
  https://review.openstack.org/#/c/326730/1

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1622545/+subscriptions