Skip to content

Assertion failed: (*card_word)==0 in DynamicGenerics tests #76801

Closed
Closed
Assertion failed: (*card_word)==0 in DynamicGenerics tests#76801
Assignees
cshungMaoni0
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabarea-GC-coreclrblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
8.0.0

Description

@jkotas
jkotas
opened on Oct 10, 2022
Failed test: nativeaot_SmokeTests::_DynamicGenerics_DynamicGenerics_DynamicGenerics_cmd (nativeaot\\SmokeTests\\DynamicGenerics\\DynamicGenerics\\DynamicGenerics.cmd)
Assertion failed: (*card_word)==0, file D:\a\_work\1\s\src\coreclr\gc\gc.cpp, line 13092

Failed in #76799 (windows.x64.Checked). Full log: https://dev.azure.com/dnceng-public/public/_build/results?buildId=45701&view=logs&jobId=58dc7ccb-0414-5dd3-62a5-bf2e63258b7c&j=ea7d6d51-d18d-5306-47bd-e3b7db04467e&t=3c9d9518-9046-50a7-9208-2e9137da71b6

{
  "ErrorMessage": "(*card_word)==0",
  "BuildRetry": false
}

Report

Build Definition Step Name Console log Pull Request
58837 dotnet/runtime Run tests in single file mode Log #76985
59002 dotnet/runtime Run tests in single file mode Log #62863
58981 dotnet/runtime Run tests in single file mode Log #77137
58748 dotnet/runtime Run tests in single file mode Log #77289
58346 dotnet/runtime Run tests in single file mode Log #77272
58445 dotnet/runtime Run tests in single file mode Log #77280
58398 dotnet/runtime Run tests in single file mode Log #76901
58362 dotnet/runtime Run tests in single file mode Log #77277
58366 dotnet/runtime Run tests in single file mode Log #77277
58323 dotnet/runtime Run tests in single file mode Log #77078
58278 dotnet/runtime Run tests in single file mode Log #77267
57679 dotnet/runtime Run tests in single file mode Log #77237
58192 dotnet/runtime Run tests in single file mode Log #73472
58181 dotnet/runtime Run tests in single file mode Log #77213
57257 dotnet/runtime Run tests in single file mode Log #77213
58029 dotnet/runtime Run tests in single file mode Log #77245
57955 dotnet/runtime Run tests in single file mode Log #77239
57923 dotnet/runtime Run tests in single file mode Log #77243
57916 dotnet/runtime Run tests in single file mode Log #77189
57869 dotnet/runtime Run tests in single file mode Log #62689
57894 dotnet/runtime Run tests in single file mode Log #76981
57856 dotnet/runtime Run tests in single file mode Log #76985
57711 dotnet/runtime Run tests in single file mode Log #77228
57651 dotnet/runtime Run tests in single file mode Log #74963
57635 dotnet/runtime Run tests in single file mode Log #74157
57589 dotnet/runtime Run tests in single file mode Log #74820
57537 dotnet/runtime Run tests in single file mode Log #77231
57502 dotnet/runtime Run tests in single file mode Log #77011
57470 dotnet/runtime Run tests in single file mode Log #76981
57458 dotnet/runtime Run tests in single file mode Log #75140
56859 dotnet/runtime Run tests in single file mode Log #77196
57079 dotnet/runtime Run tests in single file mode Log #77176
57047 dotnet/runtime Run tests in single file mode Log #77214
56766 dotnet/runtime Run tests in single file mode Log #77197
56623 dotnet/runtime Run tests in single file mode Log #77189
56287 dotnet/runtime Run tests in single file mode Log #77052
56168 dotnet/runtime Run tests in single file mode Log #77102
54722 dotnet/runtime Run tests in single file mode Log #76924
55967 dotnet/runtime Run tests in single file mode Log #70941
55952 dotnet/runtime Run tests in single file mode Log #77166
55933 dotnet/runtime Run tests in single file mode Log #77147
55914 dotnet/runtime Run tests in single file mode Log #77163
55901 dotnet/runtime Run tests in single file mode Log #74820
55803 dotnet/runtime Run tests in single file mode Log #77151
55653 dotnet/runtime Run tests in single file mode Log #77107
55265 dotnet/runtime Run tests in single file mode Log #77106
55251 dotnet/runtime Run tests in single file mode Log #77132
55243 dotnet/runtime Run tests in single file mode Log #76981
55062 dotnet/runtime Run tests in single file mode Log #75542
55028 dotnet/runtime Run tests in single file mode Log #76942
54886 dotnet/runtime Run tests in single file mode Log #73472
54750 dotnet/runtime Run tests in single file mode Log #77043
54700 dotnet/runtime Run tests in single file mode Log #77112
54551 dotnet/runtime Run tests in single file mode Log #77113
54522 dotnet/runtime Run tests in single file mode Log #77107
54457 dotnet/runtime Run tests in single file mode Log #77103
54175 dotnet/runtime Run tests in single file mode Log #77065
54303 dotnet/runtime Run tests in single file mode Log #77103
54153 dotnet/runtime Run tests in single file mode Log #77102
53820 dotnet/runtime Run tests in single file mode Log #77065
53751 dotnet/runtime Run tests in single file mode Log #77080
53655 dotnet/runtime Run tests in single file mode Log #76779
53449 dotnet/runtime Run tests in single file mode Log #77056
53287 dotnet/runtime Run tests in single file mode Log #77067
53170 dotnet/runtime Run tests in single file mode Log #77062
52849 dotnet/runtime Run tests in single file mode Log #76851
52807 dotnet/runtime Run tests in single file mode Log #77052
52590 dotnet/runtime Run tests in single file mode Log #77042
52728 dotnet/runtime Run tests in single file mode Log #70941
52655 dotnet/runtime Run tests in single file mode Log #75140
52585 dotnet/runtime Run tests in single file mode Log #77003
52413 dotnet/runtime Run tests in single file mode Log #76904
52315 dotnet/runtime Run tests in single file mode Log #77035
52242 dotnet/runtime Run tests in single file mode Log #76997
52128 dotnet/runtime Run tests in single file mode Log #77027
52118 dotnet/runtime Run tests in single file mode Log #76985
51963 dotnet/runtime Run tests in single file mode Log #76981
50357 dotnet/runtime Run tests in single file mode Log #76971
50156 dotnet/runtime Run tests in single file mode Log #76950
51782 dotnet/runtime Run tests in single file mode Log #68994
51604 dotnet/runtime Run tests in single file mode Log #76890
51454 dotnet/runtime Run tests in single file mode Log #77003
51463 dotnet/runtime Run tests in single file mode Log #76981
51434 dotnet/runtime Run tests in single file mode Log #77011
51394 dotnet/runtime Run tests in single file mode Log #73472
51277 dotnet/runtime Run tests in single file mode Log #76811
51266 dotnet/runtime Run tests in single file mode Log #76848
50960 dotnet/runtime Run tests in single file mode Log #76985
50629 dotnet/runtime Run tests in single file mode Log #76972
50557 dotnet/runtime Run tests in single file mode Log #74963
50518 dotnet/runtime Run tests in single file mode Log #76975
50008 dotnet/runtime Run tests in single file mode Log #76941
49938 dotnet/runtime Run tests in single file mode Log #74157
49860 dotnet/runtime Run tests in single file mode Log #76883
49784 dotnet/runtime Run tests in single file mode Log #76933
49747 dotnet/runtime Run tests in single file mode Log #76848
49477 dotnet/runtime Run tests in single file mode Log #76901
49169 dotnet/runtime Run tests in single file mode Log #62863
48098 dotnet/runtime Run tests in single file mode Log #76339
48968 dotnet/runtime Run tests in single file mode Log #76705
Displaying 100 of 125 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
3 63 125

Activity

  1. jkotas
    added
    area-GC-coreclr
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Oct 10, 2022
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Oct 10, 2022

  4. jkotas commented on Oct 10, 2022

    @jkotas
    jkotas
    on Oct 10, 2022
    MemberAuthor

    @dotnet/gc I suspect that this may be related to regions and frozen objects. Could you please review verify_card_bundles logic for any obvious problems?

  5. EgorBo commented on Oct 10, 2022

    @EgorBo
    EgorBo
    on Oct 10, 2022
    Member

    I assume it started to appear after #76737 ?

  6. jkotas
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Oct 10, 2022
  7. mangod9
    assigned
    Maoni0
    on Oct 11, 2022

  8. mangod9 commented on Oct 11, 2022

    @mangod9
    mangod9
    on Oct 11, 2022
    Member

    Assigning to Maoni assuming its related to enabling regions on NativeAOT?

  9. mangod9
    removed
    untriagedNew issue has not been triaged by the area owner
    on Oct 11, 2022
  10. mangod9
    added this to the 8.0.0 milestone on Oct 11, 2022
  11. Maoni0
    assigned
    cshung
    on Oct 11, 2022

  12. Maoni0 commented on Oct 11, 2022

    @Maoni0
    Maoni0
    on Oct 11, 2022
    Member

    Andrew is looking at this.

  13. trylek commented on Oct 13, 2022

    @trylek
    trylek
    on Oct 13, 2022
    Member

    I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

    cmdLine:D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
Test Harness Exitcode is : -100

    A possible explanation is that in my testing the assertion failure happened very close to the 30 minute timeout for the test.

  14. cshung commented on Oct 13, 2022

    @cshung
    cshung
    on Oct 13, 2022
    Contributor

    I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

    cmdLine:D:\a_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
    Test Harness Exitcode is : -100
    A possible explanation is that in my testing the assertion failure happened very close to the 30-minute timeout for the test.

    I noticed that when the assertion happened on my machine, it stuck at a dialog box, so that might explain the timeout.

  15. trylek commented on Oct 13, 2022

    @trylek
    trylek
    on Oct 13, 2022
    Member

    Thanks Andrew, I guess that explains a lot, in CoreCLR testing we're going a long way towards suppressing all such dialogs by disabling WER UI and by other means, it might be the case that the relatively recent NativeAOT testing is still missing some of these measures.

  16. cshung commented on Oct 15, 2022

    @cshung
    cshung
    on Oct 15, 2022
    Contributor

    Still stuck with this one, but I found something interesting.

    It appears that when the assertion happens, the line being printed is always this:
    Running Test: B282745.testMDArrayWith3Dimensions

  17. jkotas commented on Oct 15, 2022

    @jkotas
    jkotas
    on Oct 15, 2022
    MemberAuthor

    NativeAOT on Windows is building with hardware write watch currently. It does not build with FEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP and FEATURE_MANUALLY_MANAGED_CARD_BUNDLES defined. Can this config difference lead to this assertion failure?

  18. Maoni0 commented on Oct 18, 2022

    @Maoni0
    Maoni0
    on Oct 18, 2022
    Member

    I was surprised to see that these 2 defines are not defined for AoT. it does expose a bug with regions (when we make used larger for regions we are not using the OS writewatch to watch the extended portion of the card table so we are not getting updated from that part), however, I care more about making AoT have these 2 defined on windows. @cshung will open an issue for AoT folks. we'd like to get rid of the code that's still using the OS writewatch completely.

  19. cshung commented on Oct 22, 2022

    @cshung
    cshung
    on Oct 22, 2022
    Contributor

    Before #77224, I am able to reproduce the bug locally in a few minutes. After the fix, I am unable to repro it locally for 4 hours. Closing this issue opportunistically to see if that repro again. Feel free to reopen if the bug repros.

  20. cshung
    closed this as completedon Oct 22, 2022
  21. ghost locked as resolved and limited conversation to collaborators on Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Known Build ErrorUse this to report build issues in the .NET Helix tabarea-GC-coreclrblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions