[wasm][aot] System.Text.Json tests fail while linking due to OOM #61524

Activity

1. ghost added

   area-Infrastructure-mono

   on Nov 12, 2021
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Nov 12, 2021
4. radical

   added

   arch-wasmWebAssembly architectureWebAssembly architecture

   on Nov 12, 2021
6. lewing commented on Nov 14, 2021

   

   lewing

   on Nov 14, 2021

   Member

   More actions

   /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]
   /datadisks/disk1/work/BD520A30/p/build/wasm/WasmApp.Native.targets(374,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/BD520A30/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. 
   

   This is the VM killing the test because it ran out of memory.

   cc @steveisok
7. lewing

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Nov 14, 2021
8. lewing

   added this to the 7.0.0 milestone on Nov 14, 2021
9. lewing commented on Nov 14, 2021

   

   lewing

   on Nov 14, 2021

   Member

   More actions

   cc @ilyas1974
10. ericstj

    added

    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

    on Jan 7, 2022
11. ericstj commented on Jan 7, 2022

    

    ericstj

    on Jan 7, 2022

    Member

    More actions

    Still happening in release/6.0. https://dev.azure.com/dnceng/public/_build/results?buildId=1541389&view=ms.vss-test-web.build-test-results-tab&runId=43596408&resultId=158013&paneView=debug

      Linking with emcc. This may take a while ...
       "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/AF620989/t/emscripten_u6kd2dfl.rsp
       "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
       "/datadisks/disk1/work/AF620989/p/build/emsdk/node/14.15.5_64bit/bin/node" /datadisks/disk1/work/AF620989/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/AF620989/t/tmpjx6y3suh.txt
       "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
    emcc : error : '/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/AF620989/w/A26E0953/e/publish/ProxyProjectForAOTOnHelix.proj]
    /datadisks/disk1/work/AF620989/p/build/wasm/WasmApp.Native.targets(375,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/AF620989/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/AF620989/w/A26E0953/e/publish/ProxyProjectForAOTOnHelix.proj]
    

    https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-62191-merge-b78860dd399545deaa/System.Text.Json.Tests/1/console.0337f9dc.log?sv=2019-07-07&se=2022-01-27T18%3A59%3A12Z&sr=c&sp=rl&sig=wtlCKkyFcU8KzsGfG7bVdNZqf4yTj7OvdPz3dy5M0e8%3D
12. danmoseley commented on Jan 8, 2022

    

    danmoseley

    on Jan 8, 2022

    Last edited by danmoseley

    Contributor

    More actions

    In my case #63507 (comment) it was a 1 hour timeout. The one above would seem to be as well, although the step only took 45 mins not an hour.

    Is it possible to make wasm-opt more verbose (when in CI) so we might get an idea how far it gets ( --debug piped to a file that gets uploaded? ) Or, reduce the optimization level from -Oz so it's faster? (Of course I know nothing of this stuff, just wondering how we can make it actionable..)
13. danmoseley commented on Jan 8, 2022

    

    danmoseley

    on Jan 8, 2022

    Contributor

    More actions

    Again #63527
14. steveisok commented on Jan 8, 2022

    

    steveisok

    on Jan 8, 2022

    Last edited by steveisok

    Member

    More actions

    @danmoseley I may be butchering the explanation somewhat, but STJ is making LLVM work a bit too hard (generated code size is pretty large) and sometimes we run out of memory. The only two options I know of are 1.) run outside of a docker container and see if the problem goes away or 2.) bump the memory. I had pushed for 2 outright, but we're going to have to make the case by trying 1.

    @lewing Correct me if I am wrong, but changing optimization levels I don't think will help all that much.
15. radical commented on Jan 10, 2022

    

    radical

    on Jan 10, 2022

    Member

    More actions

    Correct me if I am wrong, but changing optimization levels I don't think will help all that much.

    We tried that before, and it was helping for some time but then it started to fail again like this.
16. 3 remaining items

    Load more
17. radical

    added

    disabled-testThe test is disabled in source code against the issueThe test is disabled in source code against the issue

    wasm-aot-testWebAssembly AOT TestWebAssembly AOT Test

    on Feb 16, 2022
18. radical

    added a commit that references this issue on Feb 17, 2022

    [wasm] Disable aot tests failing due to OOM (#65413)

    ...

    2196c35
19. radical commented on Feb 17, 2022

    

    radical

    on Feb 17, 2022

    Member

    More actions

    Disabled the tests in #65413, so removing blocking-clean-ci.
20. radical

    removed

    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

    on Feb 17, 2022
21. elinor-fung commented on Feb 17, 2022

    

    elinor-fung

    on Feb 17, 2022

    Member

    More actions

    @radical I'm still seeing System.Text.Json run and fail on the latest rolling build, which has #65413. It looks like that PR only disabled that test and the System.Text.Json.SourceGeneration ones on Windows ('$(BrowserHost)' == 'Windows'), but I think we want to disable it on all platforms?
22. steveisok

    added a commit that references this issue on Mar 8, 2022

    [release/6.0][wasm] Backport skipping System.Text.Json tests for aot …

    ...

    81e3c86
23. carlossanlop

    added a commit that references this issue on Mar 10, 2022

    [release/6.0][wasm] Backport skipping System.Text.Json tests for aot …

    ...

    87a80d9
24. ghost added

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Mar 23, 2022
25. radical

    closed this as completedin #67017on Mar 25, 2022
26. radical

    added a commit that references this issue on Mar 25, 2022

    [wasm][aot] Run the high resource AOT tests on build machine itself (#…

    ...

    This issue will close once commit 286492c is merged into the 'main' branch.

    286492c
27. ghost removed

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Mar 25, 2022
28. radekdoulik

    added a commit that references this issue on Mar 30, 2022

    [wasm][aot] Run the high resource AOT tests on build machine itself (d…

    ...

    This issue will close once commit 96fdef6 is merged into the 'main' branch.

    96fdef6
29. ghost locked as resolved and limited conversation to collaborators on Apr 25, 2022