yahoo-eng-team team mailing list archive

[Matthew S <1497076@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I am trying to delete an instance (UUID 78d874ef-a9ec-4ede-9489-ff00903eb7fc), but this is failing due to corrupt data in nova's database tables.
I have no idea how this corruption arose, and I don't think it's possible to find out in order to reproduce this issue. Therefore, this bug report is not about preventing the DB corruption. Instead, it's about making Nova degrade more gracefully  in the presence of such corruption.

1. We are running Juno partially upgraded to Kilo (we are mid-way
through an in-place upgrade of Juno to Kilo).

2. Relevant log files and other data will be attached shortly

3. Steps to reproduce

3.1. Install a federated multi-cell OpenStack cloud (NeCTAR)
3.2. Use it for years, progressively upgrading it to successive OpenStack releases and meanwhile performing many operations which modify the database, one of which was (I suppose) a partially-failed creation of an instance with the given UUID
3.3. Attempt to delete the half-existing instance by executing:
nova --debug force-delete 78d874ef-a9ec-4ede-9489-ff00903eb7fc

Expected results:

- All traces of the existence of the given instance shall be removed, apart perhaps from historical data, such as records with column 'deleted' non-zero, or records in shadow tables, or log files
- The client shall receive a 200 (success) HTTP status from the server

Actual results:

- The remains of the instance (eg DB records) are not removed. I will upload partial table dumps.
- The client receives a 404. I will upload console output.
- The nova-cells service attempts to violate a DB foreign key constraint and outputs a stack backtrace to its log file, which again I will upload shortly.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1497076

Title:
  Unable to delete an instance due to foreign key violation

Status in OpenStack Compute (nova):
  New

Bug description:
  I am trying to delete an instance (UUID 78d874ef-a9ec-4ede-9489-ff00903eb7fc), but this is failing due to corrupt data in nova's database tables.
  I have no idea how this corruption arose, and I don't think it's possible to find out in order to reproduce this issue. Therefore, this bug report is not about preventing the DB corruption. Instead, it's about making Nova degrade more gracefully  in the presence of such corruption.

  1. We are running Juno partially upgraded to Kilo (we are mid-way
  through an in-place upgrade of Juno to Kilo).

  2. Relevant log files and other data will be attached shortly

  3. Steps to reproduce

  3.1. Install a federated multi-cell OpenStack cloud (NeCTAR)
  3.2. Use it for years, progressively upgrading it to successive OpenStack releases and meanwhile performing many operations which modify the database, one of which was (I suppose) a partially-failed creation of an instance with the given UUID
  3.3. Attempt to delete the half-existing instance by executing:
  nova --debug force-delete 78d874ef-a9ec-4ede-9489-ff00903eb7fc

  Expected results:

  - All traces of the existence of the given instance shall be removed, apart perhaps from historical data, such as records with column 'deleted' non-zero, or records in shadow tables, or log files
  - The client shall receive a 200 (success) HTTP status from the server

  Actual results:

  - The remains of the instance (eg DB records) are not removed. I will upload partial table dumps.
  - The client receives a 404. I will upload console output.
  - The nova-cells service attempts to violate a DB foreign key constraint and outputs a stack backtrace to its log file, which again I will upload shortly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1497076/+subscriptions