Host Activation test failures on OSX: exit code 0x86 (134) #118904

Activity

1. AndyAyersMS

   added

   blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

   Known Build ErrorUse this to report build issues in the .NET Helix tabUse this to report build issues in the .NET Helix tab

   on Aug 19, 2025
2. github-actions

   added

   needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area ownersAn area label is needed to ensure this gets routed to the appropriate area owners

   on Aug 19, 2025
3. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Aug 19, 2025
4. AndyAyersMS commented on Aug 20, 2025

   

   AndyAyersMS

   on Aug 20, 2025

   MemberAuthor

   More actions

   Also seeing failures in #118902, this time no log for one an a different assertion for the other

   https://dev.azure.com/dnceng-public/public/_build/results?buildId=1126817&view=ms.vss-test-web.build-test-results-tab

   Tracing enabled @ Tue Aug 19 21:28:46 2025 UTC
   Reading from host interface version: [0x16041101:248] to initialize policy version: [0x16041101:248]
   , but found -1 (difference of -1)
   
5. jeffschwMSFT

   added

   area-Host

   on Aug 20, 2025
6. github-project-automation

   added this to AppModelon Aug 20, 2025
7. dotnet-policy-service commented on Aug 20, 2025

   

   dotnet-policy-serviceBot

   on Aug 20, 2025 – with Dotnet Policy Service

   Contributor

   More actions

   Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
   See info in area-owners.md if you want to be subscribed.
8. elinor-fung commented on Aug 20, 2025

   

   elinor-fung

   on Aug 20, 2025

   Last edited by elinor-fung

   Member

   More actions

   Actual issues appears to be a crash (SIGABRT) - updated the error message filter. Unfortunately, we don't have dumps because the dumps were too large to upload (dotnet/arcade#14201).

   In the logs, we have the trace message from the start of hostpolicy_init_t::init - the next expected trace message would be from

   runtime/src/native/corehost/hostpolicy/hostpolicy.cpp

   Line 406 in 8f1c6e8

   trace_corehost_init(hostpolicy_init, argc, argv, location);

   
   So either somewhere in hostpolicy_init_t::init or the first part of corehost_main (not much before that trace message, so likely the init).
9. ericstj

   marked Microsoft.DotNet.CoreSetup.Test.HostActivation.DependencyResolution.LocalPath.NativeLibraries_SelfContained fails with error 0x86 (134) on macos #118945 as a duplicate of this issue on Aug 21, 2025
10. elinor-fung

    changed the title [-]Host Activation test failures on OSX[/-] [+]Host Activation test failures on OSX: exit code 0x86 (134)[/+] on Aug 21, 2025
11. kg commented on Aug 21, 2025

    

    kg

    on Aug 21, 2025

    Contributor

    More actions

    https://dev.azure.com/dnceng-public/public/_build/results?buildId=1128917&view=logs&jobId=f9191d62-d2e5-500a-3b05-adbcf0d5b394&j=f9191d62-d2e5-500a-3b05-adbcf0d5b394&t=ab342505-af40-5f81-79b2-96469c1f8ee1 is this but wasn't captured by the error message filter for some reason, I can't tell why.
12. elinor-fung

    removed

    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area ownersAn area label is needed to ensure this gets routed to the appropriate area owners

    on Aug 21, 2025
13. elinor-fung

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Sep 3, 2025
14. elinor-fung

    added this to the 11.0.0 milestone on Sep 3, 2025
15. elinor-fung commented on Sep 3, 2025

    

    elinor-fung

    on Sep 3, 2025

    Last edited by elinor-fung

    Member

    More actions

    I haven't been able to repro this on my arm64 Mac.

    I've been manually sending off to Helix (OSX.1200.Amd64.Open) one of the tests that hit this with a loop. With some extra logging, a trace message added after L102 is printed and a trace message added before L132 is not. It seems we're crashing after we exit the if block here:
    

    runtime/src/native/corehost/hostpolicy/hostpolicy_init.cpp

    Lines 96 to 103 in b9fbc0d

    	for (size_t i = 0; i < fx_count; ++i)
    	{
    	trace::verbose(_X(" name='%s', dir='%s', requested_version='%s', found_version='%s'"),
    	fx_names[i].c_str(), fx_dirs[i].c_str(), fx_requested_versions[i].c_str(), fx_found_versions[i].c_str());
    	auto fx = new fx_definition_t(fx_names[i], fx_dirs[i], fx_requested_versions[i], fx_found_versions[i]);
    	init->fx_definitions.push_back(std::unique_ptr<fx_definition_t>(fx));
    	}
    	}

    
    before we get to:

    runtime/src/native/corehost/hostpolicy/hostpolicy_init.cpp

    Line 132 in b9fbc0d

    init_host_command(input, init);
16. elinor-fung commented on Sep 5, 2025

    

    elinor-fung

    on Sep 5, 2025

    Member

    More actions

    Just based on when this started happening, I tried reverting 7195086. Without that, my 'run in a loop' repro (5000 iterations of a test that intermittently fails) stops hitting this. I don't see how that change would have caused this issue though. The only thing interesting in there is get_host_env_var (always called passing a string literal) - which I've stepped / mentally gone through a bunch of times and don't see what would be the issue:
    

    runtime/src/native/corehost/hostmisc/trace.cpp

    Lines 56 to 74 in 541ac3e

    	template<size_t NameLen>
    	bool get_host_env_var(const pal::char_t (&name)[NameLen], pal::string_t* value)
    	{
    	assert(name[NameLen - 1] == _X('\0')); // name is expected to be null-terminated
    	// DOTNET_HOST_* takes precedence
    	constexpr size_t dotnet_host_prefix_len = STRING_LENGTH("DOTNET_HOST_");
    	pal::char_t dotnet_host_name[dotnet_host_prefix_len + NameLen];
    	pal::snwprintf(dotnet_host_name, ARRAY_SIZE(dotnet_host_name), _X("DOTNET_HOST_%s"), name);
    	if (pal::getenv(dotnet_host_name, value))
    	return true;
    	// COREHOST_* for backwards compatibility
    	constexpr size_t corehost_prefix_len = STRING_LENGTH("COREHOST_");
    	pal::char_t corehost_name[corehost_prefix_len + NameLen];
    	pal::snwprintf(corehost_name, ARRAY_SIZE(corehost_name), _X("COREHOST_%s"), name);
    	return pal::getenv(corehost_name, value);
    	}
    	}
17. elinor-fung commented on Sep 5, 2025

    

    elinor-fung

    on Sep 5, 2025

    Member

    More actions

    Switching to appending to a std::string still repros the crash:

    template<size_t NameLen>
    bool get_host_env_var(const pal::char_t (&name)[NameLen], pal::string_t* value)
    {
        assert(name[NameLen - 1] == _X('\0')); // name is expected to be null-terminated
    
        // DOTNET_HOST_* takes precedence
        constexpr size_t dotnet_host_prefix_len = STRING_LENGTH("DOTNET_HOST_");
        pal::string_t dotnet_host_name = _X("DOTNET_HOST_");
        dotnet_host_name.append(name, NameLen - 1);
        if (pal::getenv(dotnet_host_name.c_str(), value))
            return true;
    
        // COREHOST_* for backwards compatibility
        constexpr size_t corehost_prefix_len = STRING_LENGTH("COREHOST_");
        pal::string_t corehost_name = _X("COREHOST_");
        corehost_name.append(name, NameLen - 1);
        return pal::getenv(corehost_name.c_str(), value);
    }

    but this does not:

    +bool get_host_env_var(const pal::char_t* name, size_t NameLen, pal::string_t* value)
    -template<size_t NameLen>
    -bool get_host_env_var(const pal::char_t (&name)[NameLen], pal::string_t* value)
    {
        assert(name[NameLen - 1] == _X('\0')); // name is expected to be null-terminated
    
        // DOTNET_HOST_* takes precedence
        constexpr size_t dotnet_host_prefix_len = STRING_LENGTH("DOTNET_HOST_");
        pal::string_t dotnet_host_name = _X("DOTNET_HOST_");
        dotnet_host_name.append(name, NameLen - 1);
        if (pal::getenv(dotnet_host_name.c_str(), value))
            return true;
    
        // COREHOST_* for backwards compatibility
        constexpr size_t corehost_prefix_len = STRING_LENGTH("COREHOST_");
        pal::string_t corehost_name = _X("COREHOST_");
        corehost_name.append(name, NameLen - 1);
        return pal::getenv(corehost_name.c_str(), value);
    }
    
    +bool get_host_env_var(const pal::char_t* name, pal::string_t* value)
    +{
    +    return get_host_env_var(name, pal::strlen(name) + 1, value);
    +}
18. elinor-fung commented on Sep 8, 2025

    

    elinor-fung

    on Sep 8, 2025

    Member

    More actions

    This stops reproing running on OSX.13.Amd64.Open instead of OSX.1200.Amd64.Open (with sources from main, 10k iterations of the test). Given macOS 12 is out of support (and the helix queues are slated to be removed, so we'll need to update anyways), I'm inclined to just update the queue.
19. elinor-fung

    closed this as completedin #119473on Sep 9, 2025
20. elinor-fung

    added a commit that references this issue on Sep 11, 2025

    Revert "Add tracing to `hostpolicy_init_t::init` (#118973)" (#119552)

    ...

    da67637
21. github-actions

    locked and limited conversation to collaborators on Oct 9, 2025