yahoo-eng-team team mailing list archive

[OpenStack Infra <1789633@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/597486
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=757983a4cfe3107ea6ffd0b416790ae23d91ef2e
Submitter: Zuul
Branch:    master

commit 757983a4cfe3107ea6ffd0b416790ae23d91ef2e
Author: Chris Dent <cdent@xxxxxxxxxxxxx>
Date:   Wed Aug 29 13:36:16 2018 +0100

    [placement] Make _ensure_aggregate context not independent
    
    The use of the independent context on _ensure_aggregate appears to
    be unnecessary. It causes file-based uses of SQLite dbs to fail
    (with database locked errors, as reported in the associated bug,
    1789633) and thus may mask issues with other databases. Adding the
    independent context manager was the result of a series of "throw
    stuff at the wall and see what sticks" patches, but it looks now
    that it is not required, and in some situations causes problems.
    
    Runs through the gate show that the behavior it was fixing (as
    described in bug 1786703) is not happening.
    
    Change-Id: I1f325d55ec256db34a4c3bbd230dcd8a91bce542
    Related-Bug: #1786703
    Closes-Bug: #1789633


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1789633

Title:
  placement aggregate handling has lock trouble with sqlite files

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Not sure if this is in itself a bug, but instead indicates that there
  are issues with aggregate handling that may show up in the real world
  under some conditions.

  The placecat tooling at https://github.com/cdent/placecat has a suite
  of automated tests that run every now and again against a docker
  container built from nova master. It uses a file-based sqlite
  database. With recent updates those tests are now failing with
  'database locked' errors (traceback below is from functional tests,
  but the one in the placecat server is the same). travis logs at https
  ://travis-ci.org/cdent/placecat/builds/421991104 but the traceback is
  not visible there

  In a nova checkout, I've changed the functional tests to use a file
  based database for sqlite and can intermittently replicate the problem
  when PUTting aggregates for a resource provider. It seems this came in
  with 2d7ed309ec4 ( https://review.openstack.org/#/c/592654/ ).


  -=-=-
         ERROR [nova.api.openstack.placement.fault_wrap] Placement API unexpected error: (sqlite3.OperationalError) database is locked (Background on this error at: http://sqlalche.me/e/e3q8)
      Traceback (most recent call last):
        File "nova/api/openstack/placement/fault_wrap.py", line 40, in __call__
          return self.application(environ, start_response)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
          resp = self.call_func(req, *args, **kw)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
          return self.func(req, *args, **kwargs)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/microversion_parse/middleware.py", line 80, in __call__
          response = req.get_response(self.application)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1313, in send
          application, catch_exc_info=False)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
          app_iter = application(self.environ, start_response)
        File "nova/api/openstack/placement/handler.py", line 213, in __call__
          return dispatch(environ, start_response, self._map)
        File "nova/api/openstack/placement/handler.py", line 150, in dispatch
          return handler(environ, start_response)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
          resp = self.call_func(req, *args, **kw)
        File "nova/api/openstack/placement/wsgi_wrapper.py", line 29, in call_func
          super(PlacementWsgify, self).call_func(req, *args, **kwargs)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
          return self.func(req, *args, **kwargs)
        File "nova/api/openstack/placement/util.py", line 191, in decorated_function
          return f(req)
        File "nova/api/openstack/placement/microversion.py", line 166, in decorated_func
          return _find_method(f, version, status_code)(req, *args, **kwargs)
        File "nova/api/openstack/placement/handlers/aggregate.py", line 131, in set_aggregates
          increment_generation=consider_generation)
        File "nova/api/openstack/placement/handlers/aggregate.py", line 72, in _set_aggregates
          aggregate_uuids, increment_generation=increment_generation)
        File "nova/api/openstack/placement/objects/resource_provider.py", line 991, in set_aggregates
          increment_generation=increment_generation)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
          return fn(*args, **kwargs)
        File "nova/api/openstack/placement/objects/resource_provider.py", line 557, in _set_aggregates
          agg_id = _ensure_aggregate(context, agg_uuid)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
          return fn(*args, **kwargs)
        File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
          self.gen.next()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1043, in _transaction_scope
          yield resource
        File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
          self.gen.next()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 653, in _session
          self.session.rollback()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
          self.force_reraise()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
          six.reraise(self.type_, self.value, self.tb)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 650, in _session
          self._end_session_transaction(self.session)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 678, in _end_session_transaction
          session.commit()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
          self.transaction.commit()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 471, in commit
          t[1].commit()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1643, in commit
          self._do_commit()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1674, in _do_commit
          self.connection._commit_impl()
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl
          self._handle_dbapi_exception(e, None, None, None, None)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
          util.raise_from_cause(newraise, exc_info)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
          reraise(type(exception), exception, tb=exc_tb, cause=cause)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl
          self.engine.dialect.do_commit(self.connection)
        File "/mnt/share/cdentsrc/nova/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 462, in do_commit
          dbapi_connection.commit()
      OperationalError: (sqlite3.OperationalError) database is locked (Background on this error at: http://sqlalche.me/e/e3q8)
          INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "PUT /resource_providers/ed83df92-38c2-4674-8aaa-774190e9d5f2/aggregates" status: 500 len: 362 microversion: 1.30
  -=-=-

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1789633/+subscriptions