yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1199179] Re: nova-manage db archive_deleted_rows throws IntegrityErrors

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mathew Odden <mrodden@xxxxxxxxxx>
Date: Wed, 17 Sep 2014 18:26:40 -0000
Reply-to: Bug 1199179 <1199179@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This was on a Grizzly cloud wit a MySQL database. It doesn't appear to
be having the issue anymore, so I think we can just close this.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1199179

Title:
  nova-manage db archive_deleted_rows throws IntegrityErrors

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  I have a cronjob that run /usr/bin/nova-manage db archive_deleted_rows
  every so often. Every time the job runs I get the same exception /
  stacktrace in the nova-manage logs about a primary key conflict.

  (Its a messy cut paste so here is the same thing on pastebin:
  http://paste.openstack.org/show/39791/ )

  2013-07-08 18:40:02.525 CRITICAL nova [req-30a84beb-ed73-49e0-af20-0727609998ce None None] (IntegrityError) (1062, "Duplicate entry '1' for key 'PRIMARY'") 'INSERT
   INTO shadow_reservations (created_at, updated_at, deleted_at, id, uuid, usage_id, project_id, resource, delta, expire, deleted) VALUES (%s, %s, %s, %s, %s, %s, %s
  , %s, %s, %s, %s)' ((datetime.datetime(2013, 6, 20, 21, 50, 18), None, datetime.datetime(2013, 6, 20, 21, 50, 19), 1L, '6367cbd2-df41-468b-ab52-b7cef7a2bab8', 13L,
   'd6c1c8f608eb4932846785b49e1c1c74', 'instances', 1L, datetime.datetime(2013, 6, 21, 21, 50, 18), 1L), (datetime.datetime(2013, 6, 20, 21, 50, 19), None, datetime.
  datetime(2013, 6, 20, 21, 50, 19), 2L, '0f5e524f-0314-4356-8b51-35cb39b64e17', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', 4096L, datetime.datetime(2013, 6, 21
  , 21, 50, 18), 2L), (datetime.datetime(2013, 6, 20, 21, 50, 19), None, datetime.datetime(2013, 6, 20, 21, 50, 19), 3L, 'b793b94f-aedc-4bba-b264-7b8c9ca0dc7e', 15L,
   'd6c1c8f608eb4932846785b49e1c1c74', 'cores', 2L, datetime.datetime(2013, 6, 21, 21, 50, 18), 3L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datet
  ime(2013, 6, 20, 21, 52, 5), 4L, '3db0fec2-6c8d-464f-ad5f-b624bb8ab67e', 13L, 'd6c1c8f608eb4932846785b49e1c1c74', 'instances', -1L, datetime.datetime(2013, 6, 21, 21, 52, 3), 4L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datetime(2013, 6, 20, 21, 52, 5), 5L, '51a8f3fa-e3e5-4865-b268-4bee1b29e151', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', -4096L, datetime.datetime(2013, 6, 21, 21, 52, 3), 5L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datetime(2013, 6, 20, 21, 52, 5), 6L, '600dcd6c-edbe-4d14-a5e4-82d8b4dfc05e', 15L, 'd6c1c8f608eb4932846785b49e1c1c74', 'cores', -2L, datetime.datetime(2013, 6, 21, 21, 52, 3), 6L), (datetime.datetime(2013, 6, 20, 21, 52, 33), None, datetime.datetime(2013, 6, 20, 21, 52, 33), 7L, 'c3f76c71-f568-4af9-9a1d-94ed338bebd7', 13L, 'd6c1c8f608eb4932846785b49e1c1c74', 'instances', 1L, datetime.datetime(2013, 6, 21, 21, 52, 33), 7L), (datetime.datetime(2013, 6, 20, 21, 52, 33), None, datetime.datetime(2013, 6, 20, 21, 52, 33), 8L, '7e8f2867-a21d-4734-bdb9-dce369fa176e', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', 4096L, datetime.datetime(2013, 6, 21, 21, 52, 33), 8L)  ... displaying 10 of 100 total bound parameter sets ...  (datetime.datetime(2013, 6, 27, 17, 56, 57), None, datetime.datetime(2013, 6, 27, 17, 56, 57), 99L, 'cf6e80da-9a82-4d26-9735-54403ba64f96', 3L, 'f258ea8877f947b3a8d32995751a9c29', 'cores', 1L, datetime.datetime(2013, 6, 28, 17, 56, 57), 99L), (datetime.datetime(2013, 6, 27, 18, 4, 17), None, datetime.datetime(2013, 6, 27, 18, 4, 20), 100L, 'bd6941b0-02ef-4b6e-ac5f-cdecef08d013', 22L, '76960cb5af8e43c4a96341b117f192fb', 'instances', -1L, datetime.datetime(2013, 6, 28, 18, 4, 17), 100L))
  2013-07-08 18:40:02.525 4707 TRACE nova Traceback (most recent call last):
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/bin/nova-manage", line 1255, in <module>
  2013-07-08 18:40:02.525 4707 TRACE nova     main()
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/bin/nova-manage", line 1247, in main
  2013-07-08 18:40:02.525 4707 TRACE nova     fn(*fn_args, **fn_kwargs)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/bin/nova-manage", line 808, in archive_deleted_rows
  2013-07-08 18:40:02.525 4707 TRACE nova     db.archive_deleted_rows(admin_context, max_rows)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 1707, in archive_deleted_rows
  2013-07-08 18:40:02.525 4707 TRACE nova     return IMPL.archive_deleted_rows(context, max_rows=max_rows)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 96, in wrapper
  2013-07-08 18:40:02.525 4707 TRACE nova     return f(*args, **kwargs)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 4783, in archive_deleted_rows
  2013-07-08 18:40:02.525 4707 TRACE nova     max_rows=max_rows - rows_archived)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 96, in wrapper
  2013-07-08 18:40:02.525 4707 TRACE nova     return f(*args, **kwargs)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 4763, in archive_deleted_rows_for_table
  2013-07-08 18:40:02.525 4707 TRACE nova     conn.execute(insert_statement, rows)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
  2013-07-08 18:40:02.525 4707 TRACE nova     params)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
  2013-07-08 18:40:02.525 4707 TRACE nova     compiled_sql, distilled_params
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
  2013-07-08 18:40:02.525 4707 TRACE nova     context)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1680, in _execute_context
  2013-07-08 18:40:02.525 4707 TRACE nova     context)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/sqlalchemy/connectors/mysqldb.py", line 55, in do_executemany
  2013-07-08 18:40:02.525 4707 TRACE nova     rowcount = cursor.executemany(statement, parameters)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 206, in executemany
  2013-07-08 18:40:02.525 4707 TRACE nova     r = r + self.execute(query, a)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  2013-07-08 18:40:02.525 4707 TRACE nova     self.errorhandler(self, exc, value)
  2013-07-08 18:40:02.525 4707 TRACE nova   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2013-07-08 18:40:02.525 4707 TRACE nova     raise errorclass, errorvalue
  2013-07-08 18:40:02.525 4707 TRACE nova IntegrityError: (IntegrityError) (1062, "Duplicate entry '1' for key 'PRIMARY'") 'INSERT INTO shadow_reservations (created_at, updated_at, deleted_at, id, uuid, usage_id, project_id, resource, delta, expire, deleted) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' ((datetime.datetime(2013, 6, 20, 21, 50, 18), None, datetime.datetime(2013, 6, 20, 21, 50, 19), 1L, '6367cbd2-df41-468b-ab52-b7cef7a2bab8', 13L, 'd6c1c8f608eb4932846785b49e1c1c74', 'instances', 1L, datetime.datetime(2013, 6, 21, 21, 50, 18), 1L), (datetime.datetime(2013, 6, 20, 21, 50, 19), None, datetime.datetime(2013, 6, 20, 21, 50, 19), 2L, '0f5e524f-0314-4356-8b51-35cb39b64e17', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', 4096L, datetime.datetime(2013, 6, 21, 21, 50, 18), 2L), (datetime.datetime(2013, 6, 20, 21, 50, 19), None, datetime.datetime(2013, 6, 20, 21, 50, 19), 3L, 'b793b94f-aedc-4bba-b264-7b8c9ca0dc7e', 15L, 'd6c1c8f608eb4932846785b49e1c1c74', 'cores', 2L, datetime.datetime(2013, 6, 21, 21, 50, 18), 3L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datetime(2013, 6, 20, 21, 52, 5), 4L, '3db0fec2-6c8d-464f-ad5f-b624bb8ab67e', 13L, 'd6c1c8f608eb4932846785b49e1c1c74', 'instances', -1L, datetime.datetime(2013, 6, 21, 21, 52, 3), 4L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datetime(2013, 6, 20, 21, 52, 5), 5L, '51a8f3fa-e3e5-4865-b268-4bee1b29e151', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', -4096L, datetime.datetime(2013, 6, 21, 21, 52, 3), 5L), (datetime.datetime(2013, 6, 20, 21, 52, 3), None, datetime.datetime(2013, 6, 20, 21, 52, 5), 6L, '600dcd6c-edbe-4d14-a5e4-82d8b4dfc05e', 15L, 'd6c1c8f608eb4932846785b49e1c1c74', 'cores', -2L, datetime.datetime(2013, 6, 21, 21, 52, 3), 6L), (datetime.datetime(2013, 6, 20, 21, 52, 33), None, datetime.datetime(2013, 6, 20, 21, 52, 33), 7L, 'c3f76c71-f568-4af9-9a1d-94ed338bebd7', 13L, 'd6c1c8f608eb4932846785b49e1c1c74', 'instances', 1L, datetime.datetime(2013, 6, 21, 21, 52, 33), 7L), (datetime.datetime(2013, 6, 20, 21, 52, 33), None, datetime.datetime(2013, 6, 20, 21, 52, 33), 8L, '7e8f2867-a21d-4734-bdb9-dce369fa176e', 14L, 'd6c1c8f608eb4932846785b49e1c1c74', 'ram', 4096L, datetime.datetime(2013, 6, 21, 21, 52, 33), 8L)  ... displaying 10 of 100 total bound parameter sets ...  (datetime.datetime(2013, 6, 27, 17, 56, 57), None, datetime.datetime(2013, 6, 27, 17, 56, 57), 99L, 'cf6e80da-9a82-4d26-9735-54403ba64f96', 3L, 'f258ea8877f947b3a8d32995751a9c29', 'cores', 1L, datetime.datetime(2013, 6, 28, 17, 56, 57), 99L), (datetime.datetime(2013, 6, 27, 18, 4, 17), None, datetime.datetime(2013, 6, 27, 18, 4, 20), 100L, 'bd6941b0-02ef-4b6e-ac5f-cdecef08d013', 22L, '76960cb5af8e43c4a96341b117f192fb', 'instances', -1L, datetime.datetime(2013, 6, 28, 18, 4, 17), 100L))

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1199179/+subscriptions