Skip to content

Deferred data expansion repeatedly scans all registered tests #7474

Closed

Description

@thomhurst

Deferred data expansion scans every registered test once for each placeholder in PR #7471. Large suites with many deferred data sources can therefore spend quadratic time preparing dependencies before test execution begins.

Reproduction

Create many test methods using [MethodDataSource(..., DeferEnumeration = true)], each yielding multiple cases, and measure session preparation as the number of methods grows. No dependency attributes are required: ReplaceDeferredTest still visits every registered test for each expansion.

The scan is visible in TestDependencyResolver.cs at dd9ebe89, called once per placeholder by the session coordinator. The current main branch does not yet contain that replacement API; this follow-up applies when #7471 lands. Native prerequisite: #7409, delivered by #7471.

Proposal

Batch placeholder replacements, invalidate affected dependency edges in one pass, and rebuild lookup indexes once before resolving dependencies. Preserve completed zero-child and failed expansion outcomes. Measure preparation time and allocations before and after, and validate dependency chains in source-generated and reflection modes.

Please make deferred dependency preparation scale with the total number of tests and dependency edges rather than repeated full-suite scans.

Acceptance criteria

  • Batch deferred placeholder replacements and dependency-edge invalidation; rebuild lookup indexes once before resolving dependencies.
  • Preserve zero-child and failed expansion outcomes and deferred dependency-chain behavior in generated and reflection modes.
  • Measure preparation time and allocations before and after on suites with increasing numbers of deferred methods and cases; demonstrate that replacement no longer scans the full registered suite once per placeholder.

Activity

  1. added
    performancePerformance optimization
    enhancementNew feature or request
    P3Low: cleanup, minor enhancement, or rare edge case
    on Nov 3, 2026
  2. thomhurst commented on Nov 3, 2026

    @thomhurst
    OwnerAuthor

    Confirmed against origin/main 1f705642be3eec0a1a55034d6210f1532ec5946c: the replacement API is still confined to open PR #7471. Added native prerequisite #7409 (delivered by #7471) and explicit acceptance criteria so this follow-up is claimed after the prerequisite lands.

  3. thomhurst commented on Nov 5, 2026

    @thomhurst
    OwnerAuthor

    Closing as already implemented by merged PR #7471, commit 801c0b42c2009726a215423cf5b5eb157db6d29d; verified against current origin/main 4c6b757bd1.

    • The coordinator accumulates all expansions and calls replacement once before dependency resolution.
    • The resolver builds one replacement set, removes placeholders in one pass, clears lookup indexes once for lazy rebuilding, and invalidates dependency edges in one pass. Completed zero-child and failed placeholders retain their outcome and lose stale outgoing edges.
    • Fresh SDK 10.0.403/net10.0/macOS arm64 validation: DeferredDependencyTests passed 8/8 in generated mode and 8/8 with --reflection; TestDependencyResolverTests passed 2/2 (empty and failed outcomes). Commands: dotnet test --project tests/TUnit.TestProject/TUnit.TestProject.csproj --framework net10.0 --treenode-filter '/*/*/DeferredDependencyTests/*' (repeat with --reflection) and dotnet test --project tests/TUnit.UnitTests/TUnit.UnitTests.csproj --framework net10.0 --treenode-filter '/*/*/TestDependencyResolverTests/*'. No Engine.Tests run locally; Fix dependencies on deferred enumeration tests #7471's engine/AOT checks passed on CI.

    PR #7471 already recorded a lightweight before/after benchmark at 1,000 placeholders x 3 children: 11.536 ms / 162,344 bytes before, 0.119 ms / 139,656 bytes after. An additional isolated Release/net10.0 replacement-loop model (old algorithm from dd9ebe89, batch algorithm from merged commit) verifies increasing method and case counts:

    Methods Cases/method Before ms Batch ms Before allocated B Batch allocated B Before full-list visits Batch full-list visits
    1,000 1 3.175 0.048 96,040 73,256 1,000,000 2,000
    1,000 3 7.048 0.054 144,088 121,304 2,001,000 4,000
    1,000 10 16.859 0.081 336,136 313,352 5,504,500 11,000
    5,000 1 81.530 0.253 480,040 322,720 25,000,000 10,000
    5,000 3 136.092 0.287 720,088 562,768 50,005,000 20,000
    5,000 10 310.354 0.551 1,680,136 1,522,816 137,522,500 55,000

    These are single-process illustrative measurements of replacement loops with lightweight objects and no dependency edges, excluding suite discovery/execution and index construction; they are not end-to-end preparation benchmarks. Exact times vary with JIT and machine load. The visit counts establish that replacement now scans once per batch rather than once per placeholder; edge invalidation visits each existing edge at most once. No additional source change is needed.

    Reproduce the isolated loop measurement

    Run this as a net10.0 console project with implicit usings enabled, using dotnet run -c Release:

    using System.Diagnostics;
    // Isolated replacement-loop model of dd9ebe89 and merged 801c0b42, excluding discovery/execution.
    foreach (var count in new[] {100, 1000, 5000})
    foreach (var cases in new[] {1, 3, 10})
    foreach (var batch in new[] {false, true})
    {
        var expansions = Enumerable.Range(0, count).Select(_ => (Placeholder: new Test(), Children: Enumerable.Range(0,cases).Select(_ => new Test()).ToArray())).ToArray();
        var tests = expansions.Select(x => x.Placeholder).ToList();
        long visits = 0;
        var allocated = GC.GetAllocatedBytesForCurrentThread();
        var watch = Stopwatch.StartNew();
        if (batch)
        {
            var replaced = new HashSet<Test>();
            foreach (var (placeholder, children) in expansions) if (children.Length > 0) replaced.Add(placeholder);
            foreach (var test in tests) { visits++; if (test.Empty) test.Dependencies = []; }
            tests.RemoveAll(replaced.Contains);
            foreach (var (_, children) in expansions) tests.AddRange(children);
            foreach (var test in tests) { visits++; foreach (var dependency in test.Dependencies) if (replaced.Contains(dependency)) {test.Dependencies=[]; break;} }
        }
        else
        {
            foreach(var (placeholder, children) in expansions)
            {
                tests.Remove(placeholder); tests.AddRange(children);
                foreach(var test in tests) { visits++; if(test.Dependencies.Any(d => d == placeholder)) test.Dependencies=[]; }
            }
        }
        watch.Stop();
        allocated=GC.GetAllocatedBytesForCurrentThread()-allocated;
        Console.WriteLine($"{count},{cases},{(batch ? "batch" : "per-placeholder")},{watch.Elapsed.TotalMilliseconds:F3},{allocated},{visits}");
    }
    class Test { public bool Empty; public Test[] Dependencies = []; }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low: cleanup, minor enhancement, or rare edge caseenhancementNew feature or requestperformancePerformance optimization

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions