yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2002951] Re: OOM kills python / mysqld in various nova devstack jobs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <2002951@xxxxxxxxxxxxxxxxxx>
Date: Wed, 18 Jan 2023 16:34:38 -0000
Reply-to: Bug 2002951 <2002951@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
FWIW, I created another change that was running this test *earlier*, and
it worked :

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_362/870924/2/check/nova-
ceph-multistore/3626391/testr_results.html

That being said, this test tooked more than 181secs so I created a new
revision for knowing how it takes for creating the cached image and how
large this cached image is using the memory :

https://review.opendev.org/c/openstack/tempest/+/870913/2/tempest/api/compute/admin/test_aaa_volume.py#90

Still waiting the results but here I think we need to modify this test
to maybe not caching this way if we can, or maybe to be run differently.


** Also affects: tempest
   Importance: Undecided
       Status: New

** Also affects: glance
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2002951

Title:
  OOM kills python / mysqld in various nova devstack jobs

Status in Glance:
  New
Status in OpenStack Compute (nova):
  Confirmed
Status in tempest:
  New

Bug description:
  The following tests exited without returning a status
  and likely segfaulted or crashed Python:

          *
  tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]

  
  And in the syslog: https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77/log/controller/logs/syslog.txt#3688

  Jan 13 22:31:13 np0032729364 kernel: Out of memory: Killed process
  114509 (python) total-vm:4966188kB, anon-rss:3914748kB, file-
  rss:5080kB, shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0

  Example run:
  https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77

  I see this happening in multiple jobs in the last 10 days:
  * nova-ceph-multistore 14x
  * nova-multi-cell 1x
  * nova-next 1x

  $ logsearch log --result FAILURE --project openstack/nova --branch master --file controller/logs/syslog.txt 'kernel: Out of memory: Killed process' --days 10
  [..snip..]
  Searching logs:
  ece0cf2ce71c4a8790a0a36529dd0a8e:/home/gibi/.cache/logsearch/ece0cf2ce71c4a8790a0a36529dd0a8e/controller/logs/syslog.txt:3774:Jan 14 22:57:33 np0032733292 kernel: Out of memory: Killed process 115024 (python) total-vm:4981004kB, anon-rss:3904068kB, file-rss:5320kB, shmem-rss:0kB, UID:1002 pgtables:9376kB oom_score_adj:0

  f5aa5edd4d354c2685fc1f3e13d0ef77:/home/gibi/.cache/logsearch/f5aa5edd4d354c2685fc1f3e13d0ef77/controller/logs/syslog.txt:3688:Jan
  13 22:31:13 np0032729364 kernel: Out of memory: Killed process 114509
  (python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB,
  shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0

  1447c6274e924e068578ca260c9ac2a6:/home/gibi/.cache/logsearch/1447c6274e924e068578ca260c9ac2a6/controller/logs/syslog.txt:3824:Jan
  13 21:34:13 np0032729237 kernel: Out of memory: Killed process 114489
  (python) total-vm:4975072kB, anon-rss:3954804kB, file-rss:5312kB,
  shmem-rss:0kB, UID:1002 pgtables:9400kB oom_score_adj:0

  446a5a73b22d432295820e5b8083a2f9:/home/gibi/.cache/logsearch/446a5a73b22d432295820e5b8083a2f9/controller/logs/syslog.txt:5103:Jan
  13 10:04:25 np0032720733 kernel: Out of memory: Killed process 48920
  (mysqld) total-vm:5233384kB, anon-rss:300872kB, file-rss:0kB, shmem-
  rss:0kB, UID:116 pgtables:2652kB oom_score_adj:0

  fae1fbe258134dd8ba060cb743707247:/home/gibi/.cache/logsearch/fae1fbe258134dd8ba060cb743707247/controller/logs/syslog.txt:6686:Jan
  13 09:44:04 np0032720410 kernel: Out of memory: Killed process 47404
  (mysqld) total-vm:5208828kB, anon-rss:278080kB, file-rss:0kB, shmem-
  rss:0kB, UID:116 pgtables:2572kB oom_score_adj:0

  1bbcaa703b7d42c7a266fde3a6acca65:/home/gibi/.cache/logsearch/1bbcaa703b7d42c7a266fde3a6acca65/controller/logs/syslog.txt:3717:Jan
  13 03:41:39 np0032719591 kernel: Out of memory: Killed process 114777
  (python) total-vm:4954352kB, anon-rss:4001500kB, file-rss:5124kB,
  shmem-rss:0kB, UID:1002 pgtables:9416kB oom_score_adj:0

  7d9ca42edc5e4bdeb17be8e8045c6468:/home/gibi/.cache/logsearch/7d9ca42edc5e4bdeb17be8e8045c6468/controller/logs/syslog.txt:3828:Jan
  12 22:06:40 np0032716841 kernel: Out of memory: Killed process 114731
  (python) total-vm:4964792kB, anon-rss:4055532kB, file-rss:5072kB,
  shmem-rss:0kB, UID:1002 pgtables:9212kB oom_score_adj:0

  bcb7bcbbbb3b478586906c31c6558b13:/home/gibi/.cache/logsearch/bcb7bcbbbb3b478586906c31c6558b13/controller/logs/syslog.txt:3769:Jan
  12 20:17:35 np0032714959 kernel: Out of memory: Killed process 114973
  (python) total-vm:4971976kB, anon-rss:3855572kB, file-rss:5356kB,
  shmem-rss:0kB, UID:1002 pgtables:9696kB oom_score_adj:0

  7572c2bf5e6547c0a1fc6b0f180a2e1f:/home/gibi/.cache/logsearch/7572c2bf5e6547c0a1fc6b0f180a2e1f/controller/logs/syslog.txt:3805:Jan
  12 17:44:16 ubuntu-focal-ovh-gra1-0032713996 kernel: Out of memory:
  Killed process 114616 (python) total-vm:4974804kB, anon-rss:3949084kB,
  file-rss:5176kB, shmem-rss:0kB, UID:1002 pgtables:9604kB
  oom_score_adj:0

  aa5cf699f8d04995b43d009e55a1accd:/home/gibi/.cache/logsearch/aa5cf699f8d04995b43d009e55a1accd/controller/logs/syslog.txt:3796:Jan
  12 16:23:26 ubuntu-focal-inmotion-iad3-0032713625 kernel: Out of
  memory: Killed process 114640 (python) total-vm:4964156kB, anon-
  rss:4310768kB, file-rss:5340kB, shmem-rss:0kB, UID:1002
  pgtables:9628kB oom_score_adj:0

  8bc71a0ec0d34373bd25d4f691136084:/home/gibi/.cache/logsearch/8bc71a0ec0d34373bd25d4f691136084/controller/logs/syslog.txt:3794:Jan
  12 15:27:35 ubuntu-focal-rax-dfw-0032712709 kernel: Out of memory:
  Killed process 114830 (python) total-vm:4968664kB, anon-rss:3861940kB,
  file-rss:5140kB, shmem-rss:0kB, UID:1002 pgtables:9380kB
  oom_score_adj:0

  81d7cef2e0b240f89fcfa727304d8e8d:/home/gibi/.cache/logsearch/81d7cef2e0b240f89fcfa727304d8e8d/controller/logs/syslog.txt:3785:Jan
  12 14:50:02 ubuntu-focal-rax-ord-0032711683 kernel: Out of memory:
  Killed process 116102 (python) total-vm:4975108kB, anon-rss:4059012kB,
  file-rss:5316kB, shmem-rss:0kB, UID:1002 pgtables:9644kB
  oom_score_adj:0

  c75eb700717b4d3c9942be1385cd45bf:/home/gibi/.cache/logsearch/c75eb700717b4d3c9942be1385cd45bf/controller/logs/syslog.txt:3777:Jan
  11 21:01:17 ubuntu-focal-rax-iad-0032702258 kernel: Out of memory:
  Killed process 114917 (python) total-vm:4969648kB, anon-rss:3886448kB,
  file-rss:5236kB, shmem-rss:0kB, UID:1002 pgtables:9732kB
  oom_score_adj:0

  fa2d7bea85ad4d29acc37d78d2adb3c3:/home/gibi/.cache/logsearch/fa2d7bea85ad4d29acc37d78d2adb3c3/controller/logs/syslog.txt:3737:Jan
  09 18:25:11 ubuntu-focal-rax-ord-0032676791 kernel: Out of memory:
  Killed process 114623 (python) total-vm:4965012kB, anon-rss:3819224kB,
  file-rss:5068kB, shmem-rss:0kB, UID:1002 pgtables:9372kB
  oom_score_adj:0

  ba57680eb8de4bf2841ed5f6b2d8b3cc:/home/gibi/.cache/logsearch/ba57680eb8de4bf2841ed5f6b2d8b3cc/controller/logs/syslog.txt:3830:Jan
  09 18:18:09 ubuntu-focal-inmotion-iad3-0032676865 kernel: Out of
  memory: Killed process 114140 (python) total-vm:4963936kB, anon-
  rss:3869684kB, file-rss:5308kB, shmem-rss:0kB, UID:1002
  pgtables:8856kB oom_score_adj:0

  4e19ddc2b0064e548093ad06205a7d67:/home/gibi/.cache/logsearch/4e19ddc2b0064e548093ad06205a7d67/controller/logs/syslog.txt:3839:Jan
  09 18:07:16 ubuntu-focal-ovh-bhs1-0032676704 kernel: Out of memory:
  Killed process 114230 (python) total-vm:4974020kB, anon-rss:3923744kB,
  file-rss:5276kB, shmem-rss:0kB, UID:1002 pgtables:9396kB
  oom_score_adj:0

  Builds with matching logs 16/409:
  +----------------------------------+---------------------+----------+-----------------------------------+----------------------+
  | uuid                             | finished            | pipeline | review                            | job                  |
  +----------------------------------+---------------------+----------+-----------------------------------+----------------------+
  | ece0cf2ce71c4a8790a0a36529dd0a8e | 2023-01-14T23:17:56 | check    | https://review.opendev.org/866218 | nova-ceph-multistore |
  | f5aa5edd4d354c2685fc1f3e13d0ef77 | 2023-01-13T23:10:05 | gate     | https://review.opendev.org/869900 | nova-ceph-multistore |
  | 1447c6274e924e068578ca260c9ac2a6 | 2023-01-13T22:03:25 | gate     | https://review.opendev.org/866218 | nova-ceph-multistore |
  | 446a5a73b22d432295820e5b8083a2f9 | 2023-01-13T10:55:20 | check    | https://review.opendev.org/869950 | nova-multi-cell      |
  | fae1fbe258134dd8ba060cb743707247 | 2023-01-13T10:47:16 | check    | https://review.opendev.org/855654 | nova-next            |
  | 1bbcaa703b7d42c7a266fde3a6acca65 | 2023-01-13T04:17:19 | check    | https://review.opendev.org/867978 | nova-ceph-multistore |
  | 7d9ca42edc5e4bdeb17be8e8045c6468 | 2023-01-12T22:36:25 | check    | https://review.opendev.org/863918 | nova-ceph-multistore |
  | bcb7bcbbbb3b478586906c31c6558b13 | 2023-01-12T20:41:13 | check    | https://review.opendev.org/869900 | nova-ceph-multistore |
  | 7572c2bf5e6547c0a1fc6b0f180a2e1f | 2023-01-12T18:22:51 | check    | https://review.opendev.org/869950 | nova-ceph-multistore |
  | aa5cf699f8d04995b43d009e55a1accd | 2023-01-12T16:52:01 | check    | https://review.opendev.org/869950 | nova-ceph-multistore |
  | 8bc71a0ec0d34373bd25d4f691136084 | 2023-01-12T15:52:19 | check    | https://review.opendev.org/870012 | nova-ceph-multistore |
  | 81d7cef2e0b240f89fcfa727304d8e8d | 2023-01-12T15:39:13 | check    | https://review.opendev.org/670213 | nova-ceph-multistore |
  | c75eb700717b4d3c9942be1385cd45bf | 2023-01-11T21:30:33 | check    | https://review.opendev.org/863916 | nova-ceph-multistore |
  | fa2d7bea85ad4d29acc37d78d2adb3c3 | 2023-01-09T18:43:44 | check    | https://review.opendev.org/863918 | nova-ceph-multistore |
  | ba57680eb8de4bf2841ed5f6b2d8b3cc | 2023-01-09T18:50:43 | check    | https://review.opendev.org/863920 | nova-ceph-multistore |
  | 4e19ddc2b0064e548093ad06205a7d67 | 2023-01-09T18:37:37 | check    | https://review.opendev.org/863915 | nova-ceph-multistore |
  +----------------------------------+---------------------+----------+-----------------------------------+----------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2002951/+subscriptions
References

[Bug 2002951] [NEW] OOM kills python / mysqld in various nova devstack jobs
From: Balazs Gibizer, 2023-01-16