yahoo-eng-team team mailing list archive

[yatin <1940243@xxxxxxxxxxxxxxxxxx>]

Reopening the bz as seeing few failures(5 failures in last 15 days as per https://opensearch.logs.openstack.org) in linuxbridge and openvswitch scenario jobs:-
https://1b33868f301e2201a22c-a64bb815b8796eabf8a53948331bd878.ssl.cf5.rackcdn.com/845366/2/check/neutron-tempest-plugin-openvswitch/aa3106c/controller/logs/syslog.txt
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_82d/843649/2/gate/neutron-tempest-plugin-openvswitch-iptables_hybrid/82d4dbe/controller/logs/syslog.txt
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_052/842135/1/check/neutron-tempest-plugin-linuxbridge/0528022/controller/logs/syslog.txt
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f11/844680/1/check/neutron-tempest-plugin-linuxbridge/f111e3f/controller/logs/syslog.txt
https://bd6c3da3b1723c5f3502-97cb3b32849366f5bed744685e46b266.ssl.cf1.rackcdn.com/844860/3/check/neutron-tempest-plugin-linuxbridge/35e5065/controller/logs/syslog.txt

In all the above failures, SwapFree: 0 kB and many vms were active(using 256 mb/128 mb each) and mysql utilizing approx 10% of total memory.
Also multiple(around 10) neutron-keepalived-state-change processes utilizing approx 12-14% of memory.

Considering only seen in master jobs and with timings seems this
triggered after https://review.opendev.org/c/openstack/neutron-tempest-
plugin/+/836912 as more api tests running along with scenario tests now.

The swap used in these jobs is 1 gb, so it can be increased to some
extent like 3-4 gb to avoid these issues due to memory shortage.

** Changed in: neutron
       Status: Fix Released => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1940243

Title:
  Neutron-tempest-plugin scenario tests - oom-killer is killing mysql
  process

Status in neutron:
  Triaged

Bug description:
  It happens pretty often recently that during our scenario tests we are
  running out of memory and oom-killer is killing mysql process as it is
  number 1 in memory consumption. That is causing job's failures.

  It seems for me that it happens when there are running tests which
  uses vms with advanced image (ubuntu). Maybe we should extract those
  tests and run them as second stage with "--concurency 1"?

  Examples of failures:

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/85afc13/testr_results.html

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/9445e5f/testr_results.html

  https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron-
  tempest-plugin-scenario-linuxbridge/770722a/testr_results.html

  https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron-
  tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html

  https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/5e597ae/testr_results.html

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron-
  tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html

  https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron-
  tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html

  https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/de03f1f/testr_results.html

  https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/98bfff3/testr_results.html

  https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron-
  tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html

  
  Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1940243/+subscriptions