yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

Seen here:

http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848
/job-output.txt.gz#_2019-04-03_22_54_01_907907

The test times out and the underlying issue seems to be an
InterfaceError from pymysql:

http://paste.openstack.org/show/748901/

The thread switch issue could be related to bug 1793364.

The funny thing about this is it started spiking since April 1 when this
merged:

https://github.com/openstack/nova/commit/e5269f2fbb773953266fccbfd9df3f75e351eeb9

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.InterfaceError%3A%20(pymysql.err.InterfaceError)%20(0%2C%20'')%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

And it predominantly shows up in the limestone-regionone nodes, I'm not
sure why.

The change above that merged on 4/1 also removed skipsdist=True from
nova's tox.ini so I'm wondering if there is something on the bionic
images in those limestone nodes that causes weird issues.

I noticed in the job that failed that tox is 3.8.4 and in another job
that passed tox was 3.8.1, but looking at:

https://tox.readthedocs.io/en/latest/changelog.html

I'm not seeing anything obvious about a behavior change for python 3.6
jobs.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: gate-failure testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1823251

Title:
  Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables
  failures since April 1 2019 on limestone-regionone

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/13/621313/21/check/openstack-tox-
  py36/ba2c848/job-output.txt.gz#_2019-04-03_22_54_01_907907

  The test times out and the underlying issue seems to be an
  InterfaceError from pymysql:

  http://paste.openstack.org/show/748901/

  The thread switch issue could be related to bug 1793364.

  The funny thing about this is it started spiking since April 1 when
  this merged:

  https://github.com/openstack/nova/commit/e5269f2fbb773953266fccbfd9df3f75e351eeb9

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.InterfaceError%3A%20(pymysql.err.InterfaceError)%20(0%2C%20'')%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

  And it predominantly shows up in the limestone-regionone nodes, I'm
  not sure why.

  The change above that merged on 4/1 also removed skipsdist=True from
  nova's tox.ini so I'm wondering if there is something on the bionic
  images in those limestone nodes that causes weird issues.

  I noticed in the job that failed that tox is 3.8.4 and in another job
  that passed tox was 3.8.1, but looking at:

  https://tox.readthedocs.io/en/latest/changelog.html

  I'm not seeing anything obvious about a behavior change for python 3.6
  jobs.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1823251/+subscriptions