yahoo-eng-team team mailing list archive

[OpenStack Infra <1687027@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/612505
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2
Submitter: Zuul
Branch:    master

commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2
Author: Slawek Kaplonski <skaplons@xxxxxxxxxx>
Date:   Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests
    
    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.
    
    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.
    
    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.
    
    [1] https://review.openstack.org/#/c/610003/
    
    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1687027

Title:
  test_walk_versions tests fail with "IndexError: tuple index out of
  range" after timeout

Status in neutron:
  Fix Released

Bug description:
  http://logs.openstack.org/99/460399/1/check/gate-neutron-dsvm-
  functional-ubuntu-xenial/25de43d/testr_results.html.gz

  Traceback (most recent call last):
    File "neutron/tests/base.py", line 115, in func
      return f(self, *args, **kwargs)
    File "neutron/tests/base.py", line 115, in func
      return f(self, *args, **kwargs)
    File "neutron/tests/functional/db/test_migrations.py", line 551, in test_walk_versions
      self._migrate_up(config, engine, dest, curr, with_data=True)
    File "neutron/tests/functional/db/test_migrations.py", line 537, in _migrate_up
      migration.do_alembic_command(config, 'upgrade', dest)
    File "neutron/db/migration/cli.py", line 109, in do_alembic_command
      getattr(alembic_command, cmd)(config, *args, **kwargs)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/command.py", line 254, in upgrade
      script.run_env()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
      util.load_python_file(self.dir, 'env.py')
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
      module = load_module_py(module_id, path)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/compat.py", line 75, in load_module_py
      mod = imp.load_source(module_id, path, fp)
    File "neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
      run_migrations_online()
    File "neutron/db/migration/alembic_migrations/env.py", line 114, in run_migrations_online
      context.run_migrations()
    File "<string>", line 8, in run_migrations
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/environment.py", line 817, in run_migrations
      self.get_context().run_migrations(**kw)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/migration.py", line 323, in run_migrations
      step.migration_fn(**kw)
    File "/opt/stack/new/neutron/neutron/db/migration/alembic_migrations/versions/mitaka/expand/3894bccad37f_add_timestamp_to_base_resources.py", line 36, in upgrade
      sa.Column(column_name, sa.DateTime(), nullable=True)
    File "<string>", line 8, in add_column
    File "<string>", line 3, in add_column
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1551, in add_column
      return operations.invoke(op)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
      return fn(self, operation)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 123, in add_column
      schema=schema
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 172, in add_column
      self._exec(base.AddColumn(table_name, column, schema=schema))
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
      return conn.execute(construct, *multiparams, **params)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
      return meth(self, multiparams, params)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
      return connection._execute_ddl(self, multiparams, params)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1002, in _execute_ddl
      compiled
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
      context)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1398, in _handle_dbapi_exception
      util.raise_from_cause(newraise, exc_info)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
      context)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
      cursor.execute(statement, parameters)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
      result = self._query(query)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
      conn.query(q)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
      self._affected_rows = self._read_query_result(unbuffered=unbuffered)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
      result.read()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
      first_packet = self.connection._read_packet()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
      packet_header = self._read_bytes(4)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1021, in _read_bytes
      data = self._rfile.read(num_bytes)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/_socketio.py", line 59, in readinto
      return self._sock.recv_into(b)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 360, in recv_into
      return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 348, in _recv_loop
      self._read_trampoline()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 319, in _read_trampoline
      timeout_exc=socket.timeout("timed out"))
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 203, in _trampoline
      mark_as_closed=self._mark_as_closed)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
      return hub.switch()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
      return self.greenlet.switch()
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
      self.wait(sleep_time)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
      presult = self.do_poll(seconds)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
      return self.poll.poll(seconds)
    File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  IndexError: tuple index out of range

  It's interesting that ALL of them fail in the same job (something
  wrong with the db backend?) Counterparts for postgresql didn't fail.

  The tests in question are quite long. (Tool psql test to pass in
  55secs, while mysql failed after 200s+). We see that mysqld is the
  most intensive process during those 200secs (see dstat). Is it really
  just slow?

  Several suggestions here:
  - we should probably squash scripts, they span till Mitaka right now.
  - there are no mysqld logs in functional job. Should we add those?
  - there is no per-testcase log file for the tests, probably because the tests are special snowflakes that don't use common base test class that would set logdir.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1687027/+subscriptions