← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #89401

[Bug 1983188] [NEW] nova-manage db purge fails on large datasets

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but
in 'nova-manage db purge'. Purging a large dataset causes this failure
in Galera cluster.  Mariadb log has following error when this happens:

2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

This happens because the transaction is too large for mariadb to handle.
'nova-manage db archive_deleted_rows' works around this by limiting the
number of max_rows using --max_rows argument and checking it against
db.MAX_INT variable, we might have to do similar thing for purge


Traceback:
nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all

An error has occurred:
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge
    status_fn=status)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables
    deleted = conn.execute(delete)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context
    self._root._commit_impl(autocommit=True)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit
    dbapi_connection.commit()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
    self._read_ok_packet()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
    pkt = self._read_packet()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT')
(Background on this error at: http://sqlalche.me/e/13/e3q8)


MariaDB log has following error:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

More information on how to replicate this and workaround this using mariadb is available here
https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: nova-manage

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1983188

Title:
  nova-manage db purge fails on large datasets

Status in OpenStack Compute (nova):
  New

Bug description:
  This is similar to older bug https://bugs.launchpad.net/bugs/1543937
  but in 'nova-manage db purge'. Purging a large dataset causes this
  failure in Galera cluster.  Mariadb log has following error when this
  happens:

  2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
  2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

  This happens because the transaction is too large for mariadb to
  handle.  'nova-manage db archive_deleted_rows' works around this by
  limiting the number of max_rows using --max_rows argument and checking
  it against db.MAX_INT variable, we might have to do similar thing for
  purge

  
  Traceback:
  nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all

  An error has occurred:
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main
      ret = fn(*fn_args, **fn_kwargs)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge
      status_fn=status)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables
      deleted = conn.execute(delete)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
      return meth(self, multiparams, params)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
      return connection._execute_clauseelement(self, multiparams, params)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
      distilled_params,
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
      e, statement, parameters, cursor, context
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
      raise exception
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context
      self._root._commit_impl(autocommit=True)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl
      self._handle_dbapi_exception(e, None, None, None, None)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception
      util.raise_(newraise, with_traceback=exc_info[2], from_=e)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
      raise exception
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
      self.engine.dialect.do_commit(self.connection)
    File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit
      dbapi_connection.commit()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit
      self._read_ok_packet()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
      pkt = self._read_packet()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet
      packet.raise_for_error()
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error
      err.raise_mysql_exception(self._data)
    File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
      raise errorclass(errno, errval)
  sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT')
  (Background on this error at: http://sqlalche.me/e/13/e3q8)

  
  MariaDB log has following error:
  2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648
  2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2

  More information on how to replicate this and workaround this using mariadb is available here
  https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1983188/+subscriptions
  Thread Previous • Date Previous • • Thread Next