yahoo-eng-team team mailing list archive

[Brian Haley <1980421@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1980421

Title:
  'Socket /var/run/openvswitch/ovnnb_db.sock not found' during ovn_start

Status in devstack:
  Fix Released
Status in neutron:
  Invalid

Bug description:
  This is the same as bug 1942697 [1] and I'm opening a new bug per
  comment #2.

  I've seen this in the gate occasionally, ovn fails to start because a
  socket file it expects is not present. Here's an example trace [2]:

  [...]

  2022-06-10 06:16:20.567 | + lib/neutron_plugins/ovn_agent:start_ovn:709 :   _start_process ovn-central.service
  2022-06-10 06:16:20.569 | + lib/neutron_plugins/ovn_agent:_start_process:237 :   sudo systemctl daemon-reload
  2022-06-10 06:16:21.393 | + lib/neutron_plugins/ovn_agent:_start_process:238 :   sudo systemctl enable ovn-central.service
  2022-06-10 06:16:23.834 | + lib/neutron_plugins/ovn_agent:_start_process:239 :   sudo systemctl restart ovn-central.service
  2022-06-10 06:16:24.814 | + lib/neutron_plugins/ovn_agent:start_ovn:714 :   wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock
  2022-06-10 06:16:24.816 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:186 :   local count=0
  2022-06-10 06:16:24.818 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:24.821 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:26.152 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=1
  2022-06-10 06:16:26.156 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 1 -gt 5 ']'
  2022-06-10 06:16:26.160 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:26.163 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:27.170 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=2
  2022-06-10 06:16:27.174 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 2 -gt 5 ']'
  2022-06-10 06:16:27.178 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:27.181 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:28.188 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=3
  2022-06-10 06:16:28.192 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 3 -gt 5 ']'
  2022-06-10 06:16:28.196 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:28.200 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:29.207 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=4
  2022-06-10 06:16:29.211 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 4 -gt 5 ']'
  2022-06-10 06:16:29.215 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:29.218 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:30.225 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=5
  2022-06-10 06:16:30.229 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 5 -gt 5 ']'
  2022-06-10 06:16:30.233 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:187 :   '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']'
  2022-06-10 06:16:30.236 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:188 :   sleep 1
  2022-06-10 06:16:31.243 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:189 :   count=6
  2022-06-10 06:16:31.247 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:190 :   '[' 6 -gt 5 ']'
  2022-06-10 06:16:31.251 | + lib/neutron_plugins/ovn_agent:wait_for_sock_file:191 :   die 191 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
  2022-06-10 06:16:31.255 | + functions-common:die:264                 :   local exitcode=0
  2022-06-10 06:16:31.259 | [Call Trace]
  2022-06-10 06:16:31.259 | ./stack.sh:1277:start_ovn_services
  2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron-legacy:512:start_ovn
  2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:714:wait_for_sock_file
  2022-06-10 06:16:31.260 | /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191:die
  2022-06-10 06:16:31.266 | [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191 Socket /var/run/openvswitch/ovnnb_db.sock not found
  2022-06-10 06:16:32.272 | exit_trap: cleaning up child processes
  2022-06-10 06:16:32.275 | Error on exit

  opensearch query:

  11 hits in the last 30 days, check and gate queues, all failures

  https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_change,build_status,build_queue),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:lucene,query:'message:%20%22%5BERROR%5D%22%20AND%20message:%22Socket%20%2Fvar%2Frun%2Fopenvswitch%2Fovnnb_db.sock%20not%20found%22%20AND%20tags:devstack'),sort:!())

  [1] https://bugs.launchpad.net/devstack/+bug/1942697
  [2] https://zuul.opendev.org/t/openstack/build/435a3650d30c40668ae06a3de7efb70f/log/controller/logs/devstacklog.txt#13448

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1980421/+subscriptions