← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #87740

[Bug 1952467] [NEW] OVN cannot create it's files when build from source

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

With OVN build from source, devstack will fail on accessing
/opt/stack/data/ovn, since ovn bits will be accessing with unprivileged
user, while that directory will be created using elevated permissions.
As a result, following crash can be observed:

+ lib/neutron_plugins/ovn_agent:_start_process:227 :   sudo systemctl enable devstack@ovn-northd.service
Created symlink /etc/systemd/system/multi-user.target.wants/devstack@ovn-northd.service → /etc/systemd/system/devstack@ovn-northd.service.
+ lib/neutron_plugins/ovn_agent:_start_process:228 :   sudo systemctl restart devstack@ovn-northd.service
Job for devstack@ovn-northd.service failed because a timeout was exceeded.
See "systemctl status devstack@ovn-northd.service" and "journalctl -xe" for details.
+ lib/neutron_plugins/ovn_agent:_start_process:1 :   exit_trap
+ ./devstack/stack.sh:exit_trap:507        :   local r=1
++ ./devstack/stack.sh:exit_trap:508        :   jobs -p
+ ./devstack/stack.sh:exit_trap:508        :   jobs=76154
+ ./devstack/stack.sh:exit_trap:511        :   [[ -n 76154 ]]
+ ./devstack/stack.sh:exit_trap:511        :   [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-151741 ]]
+ ./devstack/stack.sh:exit_trap:511        :   [[ True == \T\r\u\e ]]
+ ./devstack/stack.sh:exit_trap:512        :   echo 'exit_trap: cleaning up child processes'
exit_trap: cleaning up child processes
+ ./devstack/stack.sh:exit_trap:513        :   kill 76154
+ ./devstack/stack.sh:exit_trap:517        :   '[' -f /tmp/tmp.vXTRCtpEh5 ']'
+ ./devstack/stack.sh:exit_trap:518        :   rm /tmp/tmp.vXTRCtpEh5
+ ./devstack/stack.sh:exit_trap:522        :   kill_spinner
+ ./devstack/stack.sh:kill_spinner:417     :   '[' '!' -z '' ']'
+ ./devstack/stack.sh:exit_trap:524        :   [[ 1 -ne 0 ]]
+ ./devstack/stack.sh:exit_trap:525        :   echo 'Error on exit'
Error on exit
+ ./devstack/stack.sh:exit_trap:527        :   type -p generate-subunit
+ ./devstack/stack.sh:exit_trap:528        :   generate-subunit 1637939860 1408 fail
+ ./devstack/stack.sh:exit_trap:530        :   [[ -z /opt/stack/logs ]]
+ ./devstack/stack.sh:exit_trap:533        :   /usr/bin/python3.8 /home/ubuntu/devstack/tools/worlddump.py -d /opt/stack/logs
+ ./devstack/stack.sh:exit_trap:542        :   exit 1

and status for the unit will reveal the issue:

 $ sudo systemctl status devstack@ovn-northd.service
● devstack@ovn-northd.service - Devstack devstack@ovn-northd.service
     Loaded: loaded (/etc/systemd/system/devstack@ovn-northd.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Fri 2021-11-26 15:41:08 UTC; 3min 52s ago
    Process: 108582 ExecStart=/bin/bash /usr/local/share/ovn/scripts/ovn-ctl --no-monitor start_northd (code=killed, signal=TERM)
      Tasks: 0 (limit: 19175)
     Memory: 1.6M
     CGroup: /system.slice/system-devstack.slice/devstack@ovn-northd.service

Nov 26 15:39:38 devstack bash[108640]: chown: cannot access '/usr/local/etc/ovn': No such file or directory
Nov 26 15:39:38 devstack ovsdb-server[108641]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log
Nov 26 15:39:38 devstack ovsdb-server[108643]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied
Nov 26 15:39:38 devstack ovsdb-server[108643]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable
Nov 26 15:39:38 devstack bash[108643]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable)
Nov 26 15:39:38 devstack ovn-sbctl[108647]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init
Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: start operation timed out. Terminating.
Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: Killing process 108647 (ovn-sbctl) with signal SIGKILL.
Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: Failed with result 'timeout'.
Nov 26 15:41:08 devstack systemd[1]: Failed to start Devstack devstack@ovn-northd.service.

** Affects: devstack
     Importance: Undecided
     Assignee: Roman Dobosz (roman-dobosz)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Roman Dobosz (roman-dobosz)

** Project changed: neutron => devstack

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1952467

Title:
  OVN cannot create it's files when build from source

Status in devstack:
  In Progress

Bug description:
  With OVN build from source, devstack will fail on accessing
  /opt/stack/data/ovn, since ovn bits will be accessing with
  unprivileged user, while that directory will be created using elevated
  permissions. As a result, following crash can be observed:

  + lib/neutron_plugins/ovn_agent:_start_process:227 :   sudo systemctl enable devstack@ovn-northd.service
  Created symlink /etc/systemd/system/multi-user.target.wants/devstack@ovn-northd.service → /etc/systemd/system/devstack@ovn-northd.service.
  + lib/neutron_plugins/ovn_agent:_start_process:228 :   sudo systemctl restart devstack@ovn-northd.service
  Job for devstack@ovn-northd.service failed because a timeout was exceeded.
  See "systemctl status devstack@ovn-northd.service" and "journalctl -xe" for details.
  + lib/neutron_plugins/ovn_agent:_start_process:1 :   exit_trap
  + ./devstack/stack.sh:exit_trap:507        :   local r=1
  ++ ./devstack/stack.sh:exit_trap:508        :   jobs -p
  + ./devstack/stack.sh:exit_trap:508        :   jobs=76154
  + ./devstack/stack.sh:exit_trap:511        :   [[ -n 76154 ]]
  + ./devstack/stack.sh:exit_trap:511        :   [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-151741 ]]
  + ./devstack/stack.sh:exit_trap:511        :   [[ True == \T\r\u\e ]]
  + ./devstack/stack.sh:exit_trap:512        :   echo 'exit_trap: cleaning up child processes'
  exit_trap: cleaning up child processes
  + ./devstack/stack.sh:exit_trap:513        :   kill 76154
  + ./devstack/stack.sh:exit_trap:517        :   '[' -f /tmp/tmp.vXTRCtpEh5 ']'
  + ./devstack/stack.sh:exit_trap:518        :   rm /tmp/tmp.vXTRCtpEh5
  + ./devstack/stack.sh:exit_trap:522        :   kill_spinner
  + ./devstack/stack.sh:kill_spinner:417     :   '[' '!' -z '' ']'
  + ./devstack/stack.sh:exit_trap:524        :   [[ 1 -ne 0 ]]
  + ./devstack/stack.sh:exit_trap:525        :   echo 'Error on exit'
  Error on exit
  + ./devstack/stack.sh:exit_trap:527        :   type -p generate-subunit
  + ./devstack/stack.sh:exit_trap:528        :   generate-subunit 1637939860 1408 fail
  + ./devstack/stack.sh:exit_trap:530        :   [[ -z /opt/stack/logs ]]
  + ./devstack/stack.sh:exit_trap:533        :   /usr/bin/python3.8 /home/ubuntu/devstack/tools/worlddump.py -d /opt/stack/logs
  + ./devstack/stack.sh:exit_trap:542        :   exit 1

  and status for the unit will reveal the issue:

   $ sudo systemctl status devstack@ovn-northd.service
  ● devstack@ovn-northd.service - Devstack devstack@ovn-northd.service
       Loaded: loaded (/etc/systemd/system/devstack@ovn-northd.service; enabled; vendor preset: enabled)
       Active: failed (Result: timeout) since Fri 2021-11-26 15:41:08 UTC; 3min 52s ago
      Process: 108582 ExecStart=/bin/bash /usr/local/share/ovn/scripts/ovn-ctl --no-monitor start_northd (code=killed, signal=TERM)
        Tasks: 0 (limit: 19175)
       Memory: 1.6M
       CGroup: /system.slice/system-devstack.slice/devstack@ovn-northd.service

  Nov 26 15:39:38 devstack bash[108640]: chown: cannot access '/usr/local/etc/ovn': No such file or directory
  Nov 26 15:39:38 devstack ovsdb-server[108641]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log
  Nov 26 15:39:38 devstack ovsdb-server[108643]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied
  Nov 26 15:39:38 devstack ovsdb-server[108643]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable
  Nov 26 15:39:38 devstack bash[108643]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable)
  Nov 26 15:39:38 devstack ovn-sbctl[108647]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init
  Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: start operation timed out. Terminating.
  Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: Killing process 108647 (ovn-sbctl) with signal SIGKILL.
  Nov 26 15:41:08 devstack systemd[1]: devstack@ovn-northd.service: Failed with result 'timeout'.
  Nov 26 15:41:08 devstack systemd[1]: Failed to start Devstack devstack@ovn-northd.service.

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1952467/+subscriptions
  Thread Previous • Date Previous • • Thread Next