Skip to content

Assert failure: !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" #55517

Closed

Description

@BruceForstall

System.Net.Requests.Tests and System.Security.Cryptography.Pkcs.Tests

Many jobs, x64/arm64/arm32

https://dev.azure.com/dnceng/public/_build/results?buildId=1232250&view=ms.vss-test-web.build-test-results-tab

+ grep COMPlus
COMPlus_TieredCompilation=0
COMPlus_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
COMPlus_DbgEnableMiniDump=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/C2180A37/p
----- start Mon Jul 12 08:04:14 UTC 2021 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/C2180A37/p/dotnet exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/C2180A37/w/B3D40A47/e /datadisks/disk1/work/C2180A37/w/B3D40A47/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 334 of 353 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"

Assert failure(PID 24366 [0x00005f2e], Thread: 24381 [0x5f3d]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
    Image: /datadisks/disk1/work/C2180A37/p/dotnet

Gathering state for process 24366 dotnet
Crashing thread 00005f3d signal 00000006
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.24366.dmp
Written 271179776 bytes (66206 pages) to core file
Dump successfully written
./RunTests.sh: line 162: 24366 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/C2180A37/w/B3D40A47/e
----- end Mon Jul 12 08:04:38 UTC 2021 ----- exit code 134 ----------------------------------------------------------

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Jul 12, 2021
  2. BruceForstall commented on Jul 12, 2021

    @BruceForstall
    ContributorAuthor
  3. mangod9 commented on Jul 12, 2021

    @mangod9
    Member

    If this is a consistent failure for specific test category, might make sense for the owners to investigate if there is a smaller repro and if any allocations/pinvokes are causing a corruption.

  4. mangod9 commented on Jul 12, 2021

    @mangod9
    Member

    @karelz can some from system.net please take an initial look at the failures to determine if they might be corruption due to pinvokes here?

  5. mangod9 commented on Jul 13, 2021

    @mangod9
    Member

    Adding @janvorli as well in case this is related to the w^x work, assuming this started happening in the past day or so.

  6. janvorli commented on Jul 13, 2021

    @janvorli
    Member

    I've found the issue, I'll create a PR with a fix in a minute.

  7. karelz commented on Jul 13, 2021

    @karelz
    Member

    Thanks @janvorli! Out of curiosity, what caused the problems? Was it w^x work or some churn in Networking/crypto?

  8. janvorli commented on Jul 13, 2021

    @janvorli
    Member

    W^X work, I've forgotten to replace one call to c++ delete by an appropriate executable heap method.

  9. janvorli commented on Jul 13, 2021

    @janvorli
    Member

    And it was in a place that's executed when we allocate a data structure from multiple threads and only one of them wins so the others free what they've allocated. That's why it was intermittent and I haven't caught it locally.

  10. ghost added
    in-prThere is an active PR which will close this issue when it is merged
    on Jul 13, 2021
  11. ghost removed
    in-prThere is an active PR which will close this issue when it is merged
    on Jul 13, 2021
  12. removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 14, 2021
  13. added this to the 6.0.0 milestone on Jul 14, 2021
  14. am11 commented on Aug 5, 2021

    @am11
    Member

    @janvorli, fyi, enabling event log with DOTNET_EnableEventLog=1 (or COMPlus_EnableEventLog=1) raises this heap contamination assertion, even with a simple 'corerun helloworld.dll'. corehost has no issue with event log.

  15. ghost locked as resolved and limited conversation to collaborators on Sep 7, 2021
  16. JulieLeeMSFT commented on Apr 29, 2024

    @JulieLeeMSFT
    Member

    @janvorli, this assert happened in runtime-coreclr superpmi-collect pipeline, e.g., in System.Text.Json.Tests.JsonDocumentTests.CheckUnparsable(json: "[ 1") test. Could you please take a look?

  17. jkotas commented on Apr 30, 2024

    @jkotas
    Member

    This assert is a general sign of unmanaged heap corruption that can have many different root causes.

    This is crashing in the superpmi tool. It is different test and scenario than this issue tracked originally. I have opened a new issue for it: #101708 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions