System.Runtime.Tests Outerloop "killed" on Debian.10 and Debian.11

[karelz]

PR #58129 hit 4 times this failure:

  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 28 of 6264 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    24 Killed

Seems related to #56567

Not sure how to query Kusto to see all these failures. I bet they will be fairly common ...

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

PR #58129 hit 4 times this failure:

  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 28 of 6264 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    24 Killed

Seems related to #56567

Not sure how to query Kusto to see all these failures. I bet they will be fairly common ...

Author:	karelz
Assignees:	-
Labels:	area-System.Runtime
Milestone:	-

[buyaa-n]

Failing again in this build, the log:

  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 28 of 6279 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    24 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Thu 02 Dec 2021 10:40:04 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[ 2892.025746] 264 total pagecache pages
[ 2892.025747] 0 pages in swap cache
[ 2892.025748] Swap cache stats: add 0, delete 0, find 0/0
[ 2892.025748] Free swap  = 0kB
[ 2892.025749] Total swap = 0kB
[ 2892.025749] 2097038 pages RAM
[ 2892.025750] 0 pages HighMem/MovableOnly
[ 2892.025750] 58679 pages reserved
[ 2892.025751] 0 pages cma reserved
[ 2892.025751] 0 pages hwpoisoned
[ 2892.025752] Tasks state (memory values in pages):
[ 2892.025752] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 2892.025756] [    465]     0   465    27774      163   233472        0             0 systemd-journal
[ 2892.025758] [    468]     0   468    24428       45    94208        0             0 lvmetad
[ 2892.025760] [    476]     0   476    11040      397   118784        0         -1000 systemd-udevd
[ 2892.025761] [    575]     0   575     3067      267    69632        0             0 hv_kvp_daemon
[ 2892.025763] [    731] 62583   731    35446      110   176128        0             0 systemd-timesyn
[ 2892.025765] [    860]   100   860    20010      156   176128        0             0 systemd-network
[ 2892.025767] [    890]   101   890    17654      150   180224        0             0 systemd-resolve
[ 2892.025769] [   1096]     0  1096    20059     3262   196608        0             0 python3
[ 2892.025770] [   1104]   102  1104    66818      263   172032        0             0 rsyslogd
[ 2892.025772] [   1106]     0  1106     7938       75   102400        0             0 cron
[ 2892.025773] [   1109]     0  1109    40270       33    86016        0             0 lxcfs
[ 2892.025775] [   1110]     0  1110    42708     1954   229376        0             0 networkd-dispat
[ 2892.025777] [   1111]     0  1111    15502      142   159744        0             0 systemd-logind
[ 2892.025778] [   1114]     0  1114    72000      214   196608        0             0 accounts-daemon
[ 2892.025786] [   1118]     0  1118     7084       52    98304        0             0 atd
[ 2892.025813] [   1119]   103  1119    12514      158   143360        0          -900 dbus-daemon
[ 2892.025815] [   1134]     0  1134    27605       55   114688        0             0 irqbalance
[ 2892.025816] [   1137]     0  1137   226267     7026   303104        0          -999 containerd
[ 2892.025818] [   1145]     0  1145     4104       37    77824        0             0 agetty
[ 2892.025819] [   1146]     0  1146     3723       33    77824        0             0 agetty
[ 2892.025821] [   1150]     0  1150    72221      197   200704        0             0 polkitd
[ 2892.025823] [   1270]     0  1270     1128       17    53248        0             0 none
[ 2892.025824] [   1358]     0  1358    96652     4263   262144        0             0 python3
[ 2892.025826] [   1546]     0  1546    18076      181   192512        0         -1000 sshd
[ 2892.025827] [   2151]  1000  2151     2899       66    65536        0             0 helix.sh
[ 2892.025829] [   2616]     0  2616   284735    24690   577536        0          -500 dockerd
[ 2892.025831] [   2982]  1000  2982    44344     6855   253952        0             0 python3
[ 2892.025832] [   2986]   106  2986     7150       45   106496        0             0 uuidd
[ 2892.025834] [   2999]  1000  2999    63772     7335   274432        0             0 python3
[ 2892.025835] [   3000]  1000  3000   146774    33653   565248        0             0 python3
[ 2892.025837] [  15019]     0 15019    27991     1437    86016        0          -998 containerd-shim
[ 2892.025839] [  15038]  1000 15038      596       17    40960        0             0 helix_docker_wo
[ 2892.025841] [  15116]  1000 15116      596       17    40960        0             0 execute.sh
[ 2892.025842] [  15118]  1000 15118     1714       86    45056        0             0 bash
[ 2892.025844] [  15128]  1000 15128  2953699  1877208 15458304        0             0 dotnet
[ 2892.025845] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=7fa72756f4d363408f0bb47c6ee1a16bb96ebd7d7ca37cb2c2e2f33f8f86c62a,mems_allowed=0,global_oom,task_memcg=/docker/7fa72756f4d363408f0bb47c6ee1a16bb96ebd7d7ca37cb2c2e2f33f8f86c62a,task=dotnet,pid=15128,uid=1000
[ 2892.025890] Out of memory: Killed process 15128 (dotnet) total-vm:11814796kB, anon-rss:7508832kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15096kB oom_score_adj:0
[ 2892.209529] oom_reaper: reaped process 15128 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e

[danmoseley]

We are missing a way to get a dump when we are killed for using too much memory. #56165 is similar but for IO tests.