yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1798163] Re: Placement incomplete consumers online migration fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1798163@xxxxxxxxxxxxxxxxxx>
Date: Wed, 17 Oct 2018 12:18:28 -0000
Reply-to: Bug 1798163 <1798163@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/611115
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=730936e535e67127c76d4f27649a16d8cf05efc9
Submitter: Zuul
Branch:    master

commit 730936e535e67127c76d4f27649a16d8cf05efc9
Author: Mohammed Naser <mnaser@xxxxxxxxxxxx>
Date:   Tue Oct 16 19:54:40 2018 +0200

    Use unique consumer_id when doing online data migration
    
    If there are multiple consumers having allocations to the same
    resource provider, with different classes, it will attempt
    multiple INSERTs with the same consumer_id which is not allowed
    because of the database constraints.
    
    This patch adds a simple GROUP BY in order to ensure that the
    database server only provides us with unique values to avoid
    trying to INSERT duplicate values.
    
    Change-Id: I1acba5e65cd562472f29e354c6077f82844fa87d
    Closes-Bug: #1798163


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1798163

Title:
  Placement incomplete consumers online migration fails

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  When upgrading a cloud from Queens to Rocky, it will likely fail to
  start up properly with the following conditions (or in this case at
  least):

  - 3 instances from the same resource provider
  - 3 resource classes per allocation

  The INSERT INTO database query ends up being the following:

  INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) \
  SELECT allocations.consumer_id, 535, 536, NOW(), 0 \
  FROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \
  WHERE allocations.resource_provider_id = 4 AND consumers.id IS NULL

  However, the SQL query fails to work because the SELECT returns
  multiple duplicate consumer IDs, it breaks the allocation API as well,
  you end up with this traceback:

  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap [<snip>] Placement API unexpected error: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj): DBDuplicateEntry: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap Traceback (most recent call last):
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/fault_wrap.py", line 40, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.application(environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     resp = self.call_func(req, *args, **kw)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/microversion_parse/middleware.py", line 80, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     response = req.get_response(self.application)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     application, catch_exc_info=False)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     app_iter = application(self.environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 209, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return dispatch(environ, start_response, self._map)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 146, in dispatch
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return handler(environ, start_response)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     resp = self.call_func(req, *args, **kw)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi_wrapper.py", line 29, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     super(PlacementWsgify, self).call_func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return self.func(req, *args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/util.py", line 81, in decorated_function
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return f(req)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handlers/allocation.py", line 250, in list_for_resource_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     allocs = rp_obj.AllocationList.get_all_by_resource_provider(context, rp)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/objects/resource_provider.py", line 2063, in get_all_by_resource_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     _create_incomplete_consumers_for_provider(context, rp.id)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return fn(*args, **kwargs)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/objects/resource_provider.py", line 1923, in _create_incomplete_consumers_for_provider
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     res = ctx.session.execute(ins_stmt)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1176, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     bind, close_with_result=True).execute(clause, params or {})
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return meth(self, multiparams, params)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     return connection._execute_clauseelement(self, multiparams, params)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     compiled_sql, distilled_params
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     context)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     util.raise_from_cause(newraise, exc_info)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     context)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     cursor.execute(statement, parameters)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     self.errorhandler(self, exc, value)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap     raise errorclass, errorvalue
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap DBDuplicateEntry: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'") [SQL: u'INSERT INTO consumers (uuid, project_id, user_id, created_at, generation) SELECT allocations.consumer_id, 535, 536, %s AS anon_1, %s AS anon_2 \nFROM allocations LEFT OUTER JOIN consumers ON allocations.consumer_id = consumers.uuid \nWHERE allocations.resource_provider_id = %s AND consumers.id IS NULL'] [parameters: (datetime.datetime(2018, 10, 16, 16, 43, 57, 583715), 0, 4)] (Background on this error at: http://sqlalche.me/e/gkpj)
  2018-10-16 18:43:57.612 39463 ERROR nova.api.openstack.placement.fault_wrap

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1798163/+subscriptions
References

[Bug 1798163] [NEW] Placement incomplete consumers online migration fails
From: Mohammed Naser, 2018-10-16