yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #90998
[Bug 2002951] [NEW] OOM kills python / mysqld in various nova devstack jobs
Public bug reported:
The following tests exited without returning a status
and likely segfaulted or crashed Python:
*
tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]
And in the syslog: https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77/log/controller/logs/syslog.txt#3688
Jan 13 22:31:13 np0032729364 kernel: Out of memory: Killed process
114509 (python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB,
shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0
Example run:
https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77
I see this happening in multiple jobs in the last 10 days:
* nova-ceph-multistore 14x
* nova-multi-cell 1x
* nova-next 1x
$ logsearch log --result FAILURE --project openstack/nova --branch master --file controller/logs/syslog.txt 'kernel: Out of memory: Killed process' --days 10
[..snip..]
Searching logs:
ece0cf2ce71c4a8790a0a36529dd0a8e:/home/gibi/.cache/logsearch/ece0cf2ce71c4a8790a0a36529dd0a8e/controller/logs/syslog.txt:3774:Jan 14 22:57:33 np0032733292 kernel: Out of memory: Killed process 115024 (python) total-vm:4981004kB, anon-rss:3904068kB, file-rss:5320kB, shmem-rss:0kB, UID:1002 pgtables:9376kB oom_score_adj:0
f5aa5edd4d354c2685fc1f3e13d0ef77:/home/gibi/.cache/logsearch/f5aa5edd4d354c2685fc1f3e13d0ef77/controller/logs/syslog.txt:3688:Jan
13 22:31:13 np0032729364 kernel: Out of memory: Killed process 114509
(python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB, shmem-
rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0
1447c6274e924e068578ca260c9ac2a6:/home/gibi/.cache/logsearch/1447c6274e924e068578ca260c9ac2a6/controller/logs/syslog.txt:3824:Jan
13 21:34:13 np0032729237 kernel: Out of memory: Killed process 114489
(python) total-vm:4975072kB, anon-rss:3954804kB, file-rss:5312kB, shmem-
rss:0kB, UID:1002 pgtables:9400kB oom_score_adj:0
446a5a73b22d432295820e5b8083a2f9:/home/gibi/.cache/logsearch/446a5a73b22d432295820e5b8083a2f9/controller/logs/syslog.txt:5103:Jan
13 10:04:25 np0032720733 kernel: Out of memory: Killed process 48920
(mysqld) total-vm:5233384kB, anon-rss:300872kB, file-rss:0kB, shmem-
rss:0kB, UID:116 pgtables:2652kB oom_score_adj:0
fae1fbe258134dd8ba060cb743707247:/home/gibi/.cache/logsearch/fae1fbe258134dd8ba060cb743707247/controller/logs/syslog.txt:6686:Jan
13 09:44:04 np0032720410 kernel: Out of memory: Killed process 47404
(mysqld) total-vm:5208828kB, anon-rss:278080kB, file-rss:0kB, shmem-
rss:0kB, UID:116 pgtables:2572kB oom_score_adj:0
1bbcaa703b7d42c7a266fde3a6acca65:/home/gibi/.cache/logsearch/1bbcaa703b7d42c7a266fde3a6acca65/controller/logs/syslog.txt:3717:Jan
13 03:41:39 np0032719591 kernel: Out of memory: Killed process 114777
(python) total-vm:4954352kB, anon-rss:4001500kB, file-rss:5124kB, shmem-
rss:0kB, UID:1002 pgtables:9416kB oom_score_adj:0
7d9ca42edc5e4bdeb17be8e8045c6468:/home/gibi/.cache/logsearch/7d9ca42edc5e4bdeb17be8e8045c6468/controller/logs/syslog.txt:3828:Jan
12 22:06:40 np0032716841 kernel: Out of memory: Killed process 114731
(python) total-vm:4964792kB, anon-rss:4055532kB, file-rss:5072kB, shmem-
rss:0kB, UID:1002 pgtables:9212kB oom_score_adj:0
bcb7bcbbbb3b478586906c31c6558b13:/home/gibi/.cache/logsearch/bcb7bcbbbb3b478586906c31c6558b13/controller/logs/syslog.txt:3769:Jan
12 20:17:35 np0032714959 kernel: Out of memory: Killed process 114973
(python) total-vm:4971976kB, anon-rss:3855572kB, file-rss:5356kB, shmem-
rss:0kB, UID:1002 pgtables:9696kB oom_score_adj:0
7572c2bf5e6547c0a1fc6b0f180a2e1f:/home/gibi/.cache/logsearch/7572c2bf5e6547c0a1fc6b0f180a2e1f/controller/logs/syslog.txt:3805:Jan
12 17:44:16 ubuntu-focal-ovh-gra1-0032713996 kernel: Out of memory:
Killed process 114616 (python) total-vm:4974804kB, anon-rss:3949084kB,
file-rss:5176kB, shmem-rss:0kB, UID:1002 pgtables:9604kB oom_score_adj:0
aa5cf699f8d04995b43d009e55a1accd:/home/gibi/.cache/logsearch/aa5cf699f8d04995b43d009e55a1accd/controller/logs/syslog.txt:3796:Jan
12 16:23:26 ubuntu-focal-inmotion-iad3-0032713625 kernel: Out of memory:
Killed process 114640 (python) total-vm:4964156kB, anon-rss:4310768kB,
file-rss:5340kB, shmem-rss:0kB, UID:1002 pgtables:9628kB oom_score_adj:0
8bc71a0ec0d34373bd25d4f691136084:/home/gibi/.cache/logsearch/8bc71a0ec0d34373bd25d4f691136084/controller/logs/syslog.txt:3794:Jan
12 15:27:35 ubuntu-focal-rax-dfw-0032712709 kernel: Out of memory:
Killed process 114830 (python) total-vm:4968664kB, anon-rss:3861940kB,
file-rss:5140kB, shmem-rss:0kB, UID:1002 pgtables:9380kB oom_score_adj:0
81d7cef2e0b240f89fcfa727304d8e8d:/home/gibi/.cache/logsearch/81d7cef2e0b240f89fcfa727304d8e8d/controller/logs/syslog.txt:3785:Jan
12 14:50:02 ubuntu-focal-rax-ord-0032711683 kernel: Out of memory:
Killed process 116102 (python) total-vm:4975108kB, anon-rss:4059012kB,
file-rss:5316kB, shmem-rss:0kB, UID:1002 pgtables:9644kB oom_score_adj:0
c75eb700717b4d3c9942be1385cd45bf:/home/gibi/.cache/logsearch/c75eb700717b4d3c9942be1385cd45bf/controller/logs/syslog.txt:3777:Jan
11 21:01:17 ubuntu-focal-rax-iad-0032702258 kernel: Out of memory:
Killed process 114917 (python) total-vm:4969648kB, anon-rss:3886448kB,
file-rss:5236kB, shmem-rss:0kB, UID:1002 pgtables:9732kB oom_score_adj:0
fa2d7bea85ad4d29acc37d78d2adb3c3:/home/gibi/.cache/logsearch/fa2d7bea85ad4d29acc37d78d2adb3c3/controller/logs/syslog.txt:3737:Jan
09 18:25:11 ubuntu-focal-rax-ord-0032676791 kernel: Out of memory:
Killed process 114623 (python) total-vm:4965012kB, anon-rss:3819224kB,
file-rss:5068kB, shmem-rss:0kB, UID:1002 pgtables:9372kB oom_score_adj:0
ba57680eb8de4bf2841ed5f6b2d8b3cc:/home/gibi/.cache/logsearch/ba57680eb8de4bf2841ed5f6b2d8b3cc/controller/logs/syslog.txt:3830:Jan
09 18:18:09 ubuntu-focal-inmotion-iad3-0032676865 kernel: Out of memory:
Killed process 114140 (python) total-vm:4963936kB, anon-rss:3869684kB,
file-rss:5308kB, shmem-rss:0kB, UID:1002 pgtables:8856kB oom_score_adj:0
4e19ddc2b0064e548093ad06205a7d67:/home/gibi/.cache/logsearch/4e19ddc2b0064e548093ad06205a7d67/controller/logs/syslog.txt:3839:Jan
09 18:07:16 ubuntu-focal-ovh-bhs1-0032676704 kernel: Out of memory:
Killed process 114230 (python) total-vm:4974020kB, anon-rss:3923744kB,
file-rss:5276kB, shmem-rss:0kB, UID:1002 pgtables:9396kB oom_score_adj:0
Builds with matching logs 16/409:
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| uuid | finished | pipeline | review | job |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| ece0cf2ce71c4a8790a0a36529dd0a8e | 2023-01-14T23:17:56 | check | https://review.opendev.org/866218 | nova-ceph-multistore |
| f5aa5edd4d354c2685fc1f3e13d0ef77 | 2023-01-13T23:10:05 | gate | https://review.opendev.org/869900 | nova-ceph-multistore |
| 1447c6274e924e068578ca260c9ac2a6 | 2023-01-13T22:03:25 | gate | https://review.opendev.org/866218 | nova-ceph-multistore |
| 446a5a73b22d432295820e5b8083a2f9 | 2023-01-13T10:55:20 | check | https://review.opendev.org/869950 | nova-multi-cell |
| fae1fbe258134dd8ba060cb743707247 | 2023-01-13T10:47:16 | check | https://review.opendev.org/855654 | nova-next |
| 1bbcaa703b7d42c7a266fde3a6acca65 | 2023-01-13T04:17:19 | check | https://review.opendev.org/867978 | nova-ceph-multistore |
| 7d9ca42edc5e4bdeb17be8e8045c6468 | 2023-01-12T22:36:25 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| bcb7bcbbbb3b478586906c31c6558b13 | 2023-01-12T20:41:13 | check | https://review.opendev.org/869900 | nova-ceph-multistore |
| 7572c2bf5e6547c0a1fc6b0f180a2e1f | 2023-01-12T18:22:51 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| aa5cf699f8d04995b43d009e55a1accd | 2023-01-12T16:52:01 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| 8bc71a0ec0d34373bd25d4f691136084 | 2023-01-12T15:52:19 | check | https://review.opendev.org/870012 | nova-ceph-multistore |
| 81d7cef2e0b240f89fcfa727304d8e8d | 2023-01-12T15:39:13 | check | https://review.opendev.org/670213 | nova-ceph-multistore |
| c75eb700717b4d3c9942be1385cd45bf | 2023-01-11T21:30:33 | check | https://review.opendev.org/863916 | nova-ceph-multistore |
| fa2d7bea85ad4d29acc37d78d2adb3c3 | 2023-01-09T18:43:44 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| ba57680eb8de4bf2841ed5f6b2d8b3cc | 2023-01-09T18:50:43 | check | https://review.opendev.org/863920 | nova-ceph-multistore |
| 4e19ddc2b0064e548093ad06205a7d67 | 2023-01-09T18:37:37 | check | https://review.opendev.org/863915 | nova-ceph-multistore |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2002951
Title:
OOM kills python / mysqld in various nova devstack jobs
Status in OpenStack Compute (nova):
New
Bug description:
The following tests exited without returning a status
and likely segfaulted or crashed Python:
*
tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]
And in the syslog: https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77/log/controller/logs/syslog.txt#3688
Jan 13 22:31:13 np0032729364 kernel: Out of memory: Killed process
114509 (python) total-vm:4966188kB, anon-rss:3914748kB, file-
rss:5080kB, shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0
Example run:
https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77
I see this happening in multiple jobs in the last 10 days:
* nova-ceph-multistore 14x
* nova-multi-cell 1x
* nova-next 1x
$ logsearch log --result FAILURE --project openstack/nova --branch master --file controller/logs/syslog.txt 'kernel: Out of memory: Killed process' --days 10
[..snip..]
Searching logs:
ece0cf2ce71c4a8790a0a36529dd0a8e:/home/gibi/.cache/logsearch/ece0cf2ce71c4a8790a0a36529dd0a8e/controller/logs/syslog.txt:3774:Jan 14 22:57:33 np0032733292 kernel: Out of memory: Killed process 115024 (python) total-vm:4981004kB, anon-rss:3904068kB, file-rss:5320kB, shmem-rss:0kB, UID:1002 pgtables:9376kB oom_score_adj:0
f5aa5edd4d354c2685fc1f3e13d0ef77:/home/gibi/.cache/logsearch/f5aa5edd4d354c2685fc1f3e13d0ef77/controller/logs/syslog.txt:3688:Jan
13 22:31:13 np0032729364 kernel: Out of memory: Killed process 114509
(python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB,
shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0
1447c6274e924e068578ca260c9ac2a6:/home/gibi/.cache/logsearch/1447c6274e924e068578ca260c9ac2a6/controller/logs/syslog.txt:3824:Jan
13 21:34:13 np0032729237 kernel: Out of memory: Killed process 114489
(python) total-vm:4975072kB, anon-rss:3954804kB, file-rss:5312kB,
shmem-rss:0kB, UID:1002 pgtables:9400kB oom_score_adj:0
446a5a73b22d432295820e5b8083a2f9:/home/gibi/.cache/logsearch/446a5a73b22d432295820e5b8083a2f9/controller/logs/syslog.txt:5103:Jan
13 10:04:25 np0032720733 kernel: Out of memory: Killed process 48920
(mysqld) total-vm:5233384kB, anon-rss:300872kB, file-rss:0kB, shmem-
rss:0kB, UID:116 pgtables:2652kB oom_score_adj:0
fae1fbe258134dd8ba060cb743707247:/home/gibi/.cache/logsearch/fae1fbe258134dd8ba060cb743707247/controller/logs/syslog.txt:6686:Jan
13 09:44:04 np0032720410 kernel: Out of memory: Killed process 47404
(mysqld) total-vm:5208828kB, anon-rss:278080kB, file-rss:0kB, shmem-
rss:0kB, UID:116 pgtables:2572kB oom_score_adj:0
1bbcaa703b7d42c7a266fde3a6acca65:/home/gibi/.cache/logsearch/1bbcaa703b7d42c7a266fde3a6acca65/controller/logs/syslog.txt:3717:Jan
13 03:41:39 np0032719591 kernel: Out of memory: Killed process 114777
(python) total-vm:4954352kB, anon-rss:4001500kB, file-rss:5124kB,
shmem-rss:0kB, UID:1002 pgtables:9416kB oom_score_adj:0
7d9ca42edc5e4bdeb17be8e8045c6468:/home/gibi/.cache/logsearch/7d9ca42edc5e4bdeb17be8e8045c6468/controller/logs/syslog.txt:3828:Jan
12 22:06:40 np0032716841 kernel: Out of memory: Killed process 114731
(python) total-vm:4964792kB, anon-rss:4055532kB, file-rss:5072kB,
shmem-rss:0kB, UID:1002 pgtables:9212kB oom_score_adj:0
bcb7bcbbbb3b478586906c31c6558b13:/home/gibi/.cache/logsearch/bcb7bcbbbb3b478586906c31c6558b13/controller/logs/syslog.txt:3769:Jan
12 20:17:35 np0032714959 kernel: Out of memory: Killed process 114973
(python) total-vm:4971976kB, anon-rss:3855572kB, file-rss:5356kB,
shmem-rss:0kB, UID:1002 pgtables:9696kB oom_score_adj:0
7572c2bf5e6547c0a1fc6b0f180a2e1f:/home/gibi/.cache/logsearch/7572c2bf5e6547c0a1fc6b0f180a2e1f/controller/logs/syslog.txt:3805:Jan
12 17:44:16 ubuntu-focal-ovh-gra1-0032713996 kernel: Out of memory:
Killed process 114616 (python) total-vm:4974804kB, anon-rss:3949084kB,
file-rss:5176kB, shmem-rss:0kB, UID:1002 pgtables:9604kB
oom_score_adj:0
aa5cf699f8d04995b43d009e55a1accd:/home/gibi/.cache/logsearch/aa5cf699f8d04995b43d009e55a1accd/controller/logs/syslog.txt:3796:Jan
12 16:23:26 ubuntu-focal-inmotion-iad3-0032713625 kernel: Out of
memory: Killed process 114640 (python) total-vm:4964156kB, anon-
rss:4310768kB, file-rss:5340kB, shmem-rss:0kB, UID:1002
pgtables:9628kB oom_score_adj:0
8bc71a0ec0d34373bd25d4f691136084:/home/gibi/.cache/logsearch/8bc71a0ec0d34373bd25d4f691136084/controller/logs/syslog.txt:3794:Jan
12 15:27:35 ubuntu-focal-rax-dfw-0032712709 kernel: Out of memory:
Killed process 114830 (python) total-vm:4968664kB, anon-rss:3861940kB,
file-rss:5140kB, shmem-rss:0kB, UID:1002 pgtables:9380kB
oom_score_adj:0
81d7cef2e0b240f89fcfa727304d8e8d:/home/gibi/.cache/logsearch/81d7cef2e0b240f89fcfa727304d8e8d/controller/logs/syslog.txt:3785:Jan
12 14:50:02 ubuntu-focal-rax-ord-0032711683 kernel: Out of memory:
Killed process 116102 (python) total-vm:4975108kB, anon-rss:4059012kB,
file-rss:5316kB, shmem-rss:0kB, UID:1002 pgtables:9644kB
oom_score_adj:0
c75eb700717b4d3c9942be1385cd45bf:/home/gibi/.cache/logsearch/c75eb700717b4d3c9942be1385cd45bf/controller/logs/syslog.txt:3777:Jan
11 21:01:17 ubuntu-focal-rax-iad-0032702258 kernel: Out of memory:
Killed process 114917 (python) total-vm:4969648kB, anon-rss:3886448kB,
file-rss:5236kB, shmem-rss:0kB, UID:1002 pgtables:9732kB
oom_score_adj:0
fa2d7bea85ad4d29acc37d78d2adb3c3:/home/gibi/.cache/logsearch/fa2d7bea85ad4d29acc37d78d2adb3c3/controller/logs/syslog.txt:3737:Jan
09 18:25:11 ubuntu-focal-rax-ord-0032676791 kernel: Out of memory:
Killed process 114623 (python) total-vm:4965012kB, anon-rss:3819224kB,
file-rss:5068kB, shmem-rss:0kB, UID:1002 pgtables:9372kB
oom_score_adj:0
ba57680eb8de4bf2841ed5f6b2d8b3cc:/home/gibi/.cache/logsearch/ba57680eb8de4bf2841ed5f6b2d8b3cc/controller/logs/syslog.txt:3830:Jan
09 18:18:09 ubuntu-focal-inmotion-iad3-0032676865 kernel: Out of
memory: Killed process 114140 (python) total-vm:4963936kB, anon-
rss:3869684kB, file-rss:5308kB, shmem-rss:0kB, UID:1002
pgtables:8856kB oom_score_adj:0
4e19ddc2b0064e548093ad06205a7d67:/home/gibi/.cache/logsearch/4e19ddc2b0064e548093ad06205a7d67/controller/logs/syslog.txt:3839:Jan
09 18:07:16 ubuntu-focal-ovh-bhs1-0032676704 kernel: Out of memory:
Killed process 114230 (python) total-vm:4974020kB, anon-rss:3923744kB,
file-rss:5276kB, shmem-rss:0kB, UID:1002 pgtables:9396kB
oom_score_adj:0
Builds with matching logs 16/409:
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| uuid | finished | pipeline | review | job |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| ece0cf2ce71c4a8790a0a36529dd0a8e | 2023-01-14T23:17:56 | check | https://review.opendev.org/866218 | nova-ceph-multistore |
| f5aa5edd4d354c2685fc1f3e13d0ef77 | 2023-01-13T23:10:05 | gate | https://review.opendev.org/869900 | nova-ceph-multistore |
| 1447c6274e924e068578ca260c9ac2a6 | 2023-01-13T22:03:25 | gate | https://review.opendev.org/866218 | nova-ceph-multistore |
| 446a5a73b22d432295820e5b8083a2f9 | 2023-01-13T10:55:20 | check | https://review.opendev.org/869950 | nova-multi-cell |
| fae1fbe258134dd8ba060cb743707247 | 2023-01-13T10:47:16 | check | https://review.opendev.org/855654 | nova-next |
| 1bbcaa703b7d42c7a266fde3a6acca65 | 2023-01-13T04:17:19 | check | https://review.opendev.org/867978 | nova-ceph-multistore |
| 7d9ca42edc5e4bdeb17be8e8045c6468 | 2023-01-12T22:36:25 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| bcb7bcbbbb3b478586906c31c6558b13 | 2023-01-12T20:41:13 | check | https://review.opendev.org/869900 | nova-ceph-multistore |
| 7572c2bf5e6547c0a1fc6b0f180a2e1f | 2023-01-12T18:22:51 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| aa5cf699f8d04995b43d009e55a1accd | 2023-01-12T16:52:01 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| 8bc71a0ec0d34373bd25d4f691136084 | 2023-01-12T15:52:19 | check | https://review.opendev.org/870012 | nova-ceph-multistore |
| 81d7cef2e0b240f89fcfa727304d8e8d | 2023-01-12T15:39:13 | check | https://review.opendev.org/670213 | nova-ceph-multistore |
| c75eb700717b4d3c9942be1385cd45bf | 2023-01-11T21:30:33 | check | https://review.opendev.org/863916 | nova-ceph-multistore |
| fa2d7bea85ad4d29acc37d78d2adb3c3 | 2023-01-09T18:43:44 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| ba57680eb8de4bf2841ed5f6b2d8b3cc | 2023-01-09T18:50:43 | check | https://review.opendev.org/863920 | nova-ceph-multistore |
| 4e19ddc2b0064e548093ad06205a7d67 | 2023-01-09T18:37:37 | check | https://review.opendev.org/863915 | nova-ceph-multistore |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2002951/+subscriptions
Follow ups