← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #87736

[Bug 1952395] [NEW] Tempes jobs in the ovn-octavia-provider are broken

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Failure example:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9d/819377/2/check/ovn-
octavia-provider-tempest-release/c9db1e6/job-output.txt

2021-11-26 07:20:41.865039 | controller | + ./stack.sh:exit_trap:507                 :   local r=1
2021-11-26 07:20:41.868331 | controller | ++ ./stack.sh:exit_trap:508                 :   jobs -p
2021-11-26 07:20:41.872000 | controller | + ./stack.sh:exit_trap:508                 :   jobs=84673
2021-11-26 07:20:41.875064 | controller | + ./stack.sh:exit_trap:511                 :   [[ -n 84673 ]]
2021-11-26 07:20:41.877646 | controller | + ./stack.sh:exit_trap:511                 :   [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-070811 ]]
2021-11-26 07:20:41.880814 | controller | + ./stack.sh:exit_trap:511                 :   [[ True == \T\r\u\e ]]
2021-11-26 07:20:41.883918 | controller | + ./stack.sh:exit_trap:512                 :   echo 'exit_trap: cleaning up child processes'
2021-11-26 07:20:41.883992 | controller | exit_trap: cleaning up child processes
2021-11-26 07:20:41.887286 | controller | + ./stack.sh:exit_trap:513                 :   kill 84673
2021-11-26 07:20:41.890505 | controller | + ./stack.sh:exit_trap:517                 :   '[' -f /tmp/tmp.kOZU5nmHMT ']'
2021-11-26 07:20:41.893624 | controller | + ./stack.sh:exit_trap:518                 :   rm /tmp/tmp.kOZU5nmHMT
2021-11-26 07:20:41.897844 | controller | + ./stack.sh:exit_trap:522                 :   kill_spinner
2021-11-26 07:20:41.901167 | controller | + ./stack.sh:kill_spinner:417              :   '[' '!' -z '' ']'
2021-11-26 07:20:41.904165 | controller | + ./stack.sh:exit_trap:524                 :   [[ 1 -ne 0 ]]
2021-11-26 07:20:41.906863 | controller | + ./stack.sh:exit_trap:525                 :   echo 'Error on exit'
2021-11-26 07:20:41.906923 | controller | Error on exit
2021-11-26 07:20:41.909908 | controller | + ./stack.sh:exit_trap:527                 :   type -p generate-subunit
2021-11-26 07:20:41.912989 | controller | + ./stack.sh:exit_trap:528                 :   generate-subunit 1637910489 752 fail
2021-11-26 07:20:42.221225 | controller | + ./stack.sh:exit_trap:530                 :   [[ -z /opt/stack/logs ]]
2021-11-26 07:20:42.224703 | controller | + ./stack.sh:exit_trap:533                 :   /usr/bin/python3.8 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
2021-11-26 07:20:42.690784 | controller | + ./stack.sh:exit_trap:542                 :   exit 1
2021-11-26 07:20:42.690829 | controller | *** FINISHED ***
2021-11-26 07:20:53.478505 | controller | ERROR
2021-11-26 07:20:53.478789 | controller | {


In the ovn-northd logs there is error like:

Nov 26 07:19:11.721742 ubuntu-focal-inmotion-iad3-0027510773 bash[111405]:  * Creating empty database /opt/stack/data/ovn/ovnsb_db.db
Nov 26 07:19:11.722667 ubuntu-focal-inmotion-iad3-0027510773 bash[111446]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted
Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted
Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.110711.ctl': Operation not permitted
Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted
Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.110207.ctl': Operation not permitted
Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/db.sock': Operation not permitted
Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.snoop': Operation not permitted
Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.pid': Operation not permitted
Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch': Operation not permitted
Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt': Operation not permitted
Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server.log': Operation not permitted
Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/dstat-csv.log': Operation not permitted
Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovs-vswitchd.log': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.summary': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/archive': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server-nb.log': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811.summary.2021-11-26-070811': Operation not permitted
Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs': Operation not permitted
Nov 26 07:19:11.725814 ubuntu-focal-inmotion-iad3-0027510773 bash[111449]: chown: cannot access '/usr/local/etc/ovn': No such file or directory
Nov 26 07:19:11.727636 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111450]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log
Nov 26 07:19:11.728571 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied
Nov 26 07:19:11.728596 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable
Nov 26 07:19:11.728973 ubuntu-focal-inmotion-iad3-0027510773 bash[111452]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable)
Nov 26 07:19:11.734924 ubuntu-focal-inmotion-iad3-0027510773 ovn-sbctl[111456]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init
Nov 26 07:20:41.854729 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: start operation timed out. Terminating.
Nov 26 07:20:41.855570 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: Killing process 111456 (ovn-sbctl) with signal SIGKILL.
Nov 26 07:20:41.855656 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: Failed with result 'timeout'.
Nov 26 07:20:41.855824 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: Failed to start Devstack devstack@ovn-northd.service.

** Affects: neutron
     Importance: Critical
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: gate-failure ovn-octavia-provider tempest

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1952395

Title:
  Tempes jobs in the ovn-octavia-provider are broken

Status in neutron:
  Confirmed

Bug description:
  Failure example:

  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9d/819377/2/check/ovn-
  octavia-provider-tempest-release/c9db1e6/job-output.txt

  2021-11-26 07:20:41.865039 | controller | + ./stack.sh:exit_trap:507                 :   local r=1
  2021-11-26 07:20:41.868331 | controller | ++ ./stack.sh:exit_trap:508                 :   jobs -p
  2021-11-26 07:20:41.872000 | controller | + ./stack.sh:exit_trap:508                 :   jobs=84673
  2021-11-26 07:20:41.875064 | controller | + ./stack.sh:exit_trap:511                 :   [[ -n 84673 ]]
  2021-11-26 07:20:41.877646 | controller | + ./stack.sh:exit_trap:511                 :   [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-070811 ]]
  2021-11-26 07:20:41.880814 | controller | + ./stack.sh:exit_trap:511                 :   [[ True == \T\r\u\e ]]
  2021-11-26 07:20:41.883918 | controller | + ./stack.sh:exit_trap:512                 :   echo 'exit_trap: cleaning up child processes'
  2021-11-26 07:20:41.883992 | controller | exit_trap: cleaning up child processes
  2021-11-26 07:20:41.887286 | controller | + ./stack.sh:exit_trap:513                 :   kill 84673
  2021-11-26 07:20:41.890505 | controller | + ./stack.sh:exit_trap:517                 :   '[' -f /tmp/tmp.kOZU5nmHMT ']'
  2021-11-26 07:20:41.893624 | controller | + ./stack.sh:exit_trap:518                 :   rm /tmp/tmp.kOZU5nmHMT
  2021-11-26 07:20:41.897844 | controller | + ./stack.sh:exit_trap:522                 :   kill_spinner
  2021-11-26 07:20:41.901167 | controller | + ./stack.sh:kill_spinner:417              :   '[' '!' -z '' ']'
  2021-11-26 07:20:41.904165 | controller | + ./stack.sh:exit_trap:524                 :   [[ 1 -ne 0 ]]
  2021-11-26 07:20:41.906863 | controller | + ./stack.sh:exit_trap:525                 :   echo 'Error on exit'
  2021-11-26 07:20:41.906923 | controller | Error on exit
  2021-11-26 07:20:41.909908 | controller | + ./stack.sh:exit_trap:527                 :   type -p generate-subunit
  2021-11-26 07:20:41.912989 | controller | + ./stack.sh:exit_trap:528                 :   generate-subunit 1637910489 752 fail
  2021-11-26 07:20:42.221225 | controller | + ./stack.sh:exit_trap:530                 :   [[ -z /opt/stack/logs ]]
  2021-11-26 07:20:42.224703 | controller | + ./stack.sh:exit_trap:533                 :   /usr/bin/python3.8 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
  2021-11-26 07:20:42.690784 | controller | + ./stack.sh:exit_trap:542                 :   exit 1
  2021-11-26 07:20:42.690829 | controller | *** FINISHED ***
  2021-11-26 07:20:53.478505 | controller | ERROR
  2021-11-26 07:20:53.478789 | controller | {

  
  In the ovn-northd logs there is error like:

  Nov 26 07:19:11.721742 ubuntu-focal-inmotion-iad3-0027510773 bash[111405]:  * Creating empty database /opt/stack/data/ovn/ovnsb_db.db
  Nov 26 07:19:11.722667 ubuntu-focal-inmotion-iad3-0027510773 bash[111446]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted
  Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted
  Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.110711.ctl': Operation not permitted
  Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted
  Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.110207.ctl': Operation not permitted
  Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/db.sock': Operation not permitted
  Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.snoop': Operation not permitted
  Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.pid': Operation not permitted
  Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch': Operation not permitted
  Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt': Operation not permitted
  Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server.log': Operation not permitted
  Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/dstat-csv.log': Operation not permitted
  Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovs-vswitchd.log': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.summary': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/archive': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server-nb.log': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811.summary.2021-11-26-070811': Operation not permitted
  Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs': Operation not permitted
  Nov 26 07:19:11.725814 ubuntu-focal-inmotion-iad3-0027510773 bash[111449]: chown: cannot access '/usr/local/etc/ovn': No such file or directory
  Nov 26 07:19:11.727636 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111450]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log
  Nov 26 07:19:11.728571 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied
  Nov 26 07:19:11.728596 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable
  Nov 26 07:19:11.728973 ubuntu-focal-inmotion-iad3-0027510773 bash[111452]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable)
  Nov 26 07:19:11.734924 ubuntu-focal-inmotion-iad3-0027510773 ovn-sbctl[111456]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init
  Nov 26 07:20:41.854729 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: start operation timed out. Terminating.
  Nov 26 07:20:41.855570 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: Killing process 111456 (ovn-sbctl) with signal SIGKILL.
  Nov 26 07:20:41.855656 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: devstack@ovn-northd.service: Failed with result 'timeout'.
  Nov 26 07:20:41.855824 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: Failed to start Devstack devstack@ovn-northd.service.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1952395/+subscriptions
  Thread Previous • Date Previous • • Thread Next