yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1940243@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

It happens pretty often recently that during our scenario tests we are
running out of memory and oom-killer is killing mysql process as it is
number 1 in memory consumption. That is causing job's failures.

It seems for me that it happens when there are running tests which uses
vms with advanced image (ubuntu). Maybe we should extract those tests
and run them as second stage with "--concurency 1"?

Examples of failures:

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron-
tempest-plugin-scenario-openvswitch-
iptables_hybrid/85afc13/testr_results.html

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron-
tempest-plugin-scenario-openvswitch-
iptables_hybrid/9445e5f/testr_results.html

https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron-
tempest-plugin-scenario-linuxbridge/770722a/testr_results.html

https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron-
tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html

https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron-
tempest-plugin-scenario-openvswitch-
iptables_hybrid/5e597ae/testr_results.html

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron-
tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html

https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron-
tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html

https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron-
tempest-plugin-scenario-openvswitch-
iptables_hybrid/de03f1f/testr_results.html

https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron-
tempest-plugin-scenario-openvswitch-
iptables_hybrid/98bfff3/testr_results.html

https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron-
tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html


Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1940243

Title:
  Neutron-tempest-plugin scenario tests - oom-killer is killing mysql
  process

Status in neutron:
  Confirmed

Bug description:
  It happens pretty often recently that during our scenario tests we are
  running out of memory and oom-killer is killing mysql process as it is
  number 1 in memory consumption. That is causing job's failures.

  It seems for me that it happens when there are running tests which
  uses vms with advanced image (ubuntu). Maybe we should extract those
  tests and run them as second stage with "--concurency 1"?

  Examples of failures:

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/85afc13/testr_results.html

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/9445e5f/testr_results.html

  https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron-
  tempest-plugin-scenario-linuxbridge/770722a/testr_results.html

  https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron-
  tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html

  https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/5e597ae/testr_results.html

  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron-
  tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html

  https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron-
  tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html

  https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/de03f1f/testr_results.html

  https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron-
  tempest-plugin-scenario-openvswitch-
  iptables_hybrid/98bfff3/testr_results.html

  https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron-
  tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html

  
  Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1940243/+subscriptions