yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1798163] Re: Placement incomplete consumers online migration fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 16 Oct 2018 18:37:48 -0000
Reply-to: Bug 1798163 <1798163@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
   Importance: Undecided => Critical

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Changed in: nova
     Assignee: Matt Riedemann (mriedem) => Mohammed Naser (mnaser)

** Changed in: nova/rocky
       Status: New => Confirmed

** Changed in: nova/rocky
   Importance: Undecided => Critical

** Tags added: placement upgrade

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1798163

Title:
  Placement incomplete consumers online migration fails

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  When upgrading a cloud from Queens to Rocky, it will likely fail to
  start up properly with the following conditions (or in this case at
  least):

  - 3 instances from the same resource provider
  - 3 resource classes per allocation

  The INSERT INTO database query ends up being the following:

  INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) \
  SELECT allocations.consumer_id, 535, 536, NOW(), 0 \
  FROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \
  WHERE allocations.resource_provider_id = 4 AND consumers.id IS NULL

  However, the SQL query fails to work because the SELECT returns
  multiple duplicate consumer IDs, it breaks the allocation API as well,
  you end up with this traceback:

  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap [<snip>] Placement API unexpected error: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj): DBDuplicateEntry: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap Traceback (most recent call last):
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/fault_wrap.py", line 40, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.application(environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     resp = self.call_func(req, *args, **kw)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/microversion_parse/middleware.py", line 80, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     response = req.get_response(self.application)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     application, catch_exc_info=False)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     app_iter = application(self.environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 209, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return dispatch(environ, start_response, self._map)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 146, in dispatch
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return handler(environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     resp = self.call_func(req, *args, **kw)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi_wrapper.py", line 29, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     super(PlacementWsgify, self).call_func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/util.py", line 81, in decorated_function
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return f(req)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handlers/allocation.py", line 250, in list_for_resource_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     allocs = rp_obj.AllocationList.get_all_by_resource_provider(context, rp)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/objects/resource_provider.py", line 2063, in get_all_by_resource_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     _create_incomplete_consumers_for_provider(context, rp.id)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return fn(*args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/objects/resource_provider.py", line 1923, in _create_incomplete_consumers_for_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     res = ctx.session.execute(ins_stmt)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1176, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     bind, close_with_result=True).execute(clause, params or {})
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return meth(self, multiparams, params)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return connection._execute_clauseelement(self, multiparams, params)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     compiled_sql, distilled_params
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     context)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     util.raise_from_cause(newraise, exc_info)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     context)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     cursor.execute(statement, parameters)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     self.errorhandler(self, exc, value)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     raise errorclass, errorvalue
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap DBDuplicateEntry: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1798163/+subscriptions
References

[Bug 1798163] [NEW] Placement incomplete consumers online migration fails
From: Mohammed Naser, 2018-10-16