Skip to content

Bug: dfdeedle-com-tests-static-seam-race-under-parallel-load #803

Closed as not planned

Description

@drmoisan
  • Work Mode: full-bug

Summary

UtilitiesCS.Test.Extensions.DfDeedle_COM_Tests.GetEmailDataInViewAsync_SeparatesTableSnapshotFromDataFrameTransform
fails nondeterministically in CI with a NullReferenceException. The same commit, re-run with no code
change whatsoever, passes. The test class modifies static seams, so the most probable mechanism is a
test-isolation race that additional parallel load in the assembly makes reachable.

Environment

  • OS/version: GitHub Actions windows runner, CI workflow, mstest-coverage job
  • Python version: not applicable; this is C# in UtilitiesCS.Test
  • Command/flags used: the repository mstest-coverage job over the full suite
  • Data source or fixture: mocked Table and MAPIFolder objects inside the test class

Steps to Reproduce

  1. Run the full MSTest suite in CI on a branch that adds test classes to the UtilitiesCS.Test assembly.
  2. Observe the run intermittently reporting 1 failed of 7108.
  3. Re-run the same job against the identical head commit and observe 0 failed.

Expected Behavior

The test passes deterministically, independently of how many other classes are present in the assembly
and of the order in which they execute.

Actual Behavior

Intermittent failure. Observed on pull request 802 at head b213234:
the first mstest-coverage attempt reported Total tests 7108, Passed 7107, Failed 1, and the re-run of
that same job at the same head commit reported all checks passing.

The failing assertion is not an assertion at all but a NullReferenceException thrown inside production
code at UtilitiesCS/Extensions/DfDeedle.cs line 186, where a logging call dereferences
tableSnapshot.Item1 without a null check. In the failing run the awaited EtlAsync result carried a null
first element.

Logs / Screenshots

  • Attached minimal logs or screenshot
  • Snippet: System.NullReferenceException at UtilitiesCS.DfDeedle.GetEmailDataInViewAsync,
    UtilitiesCS/Extensions/DfDeedle.cs line 186, reached from the named test method.

Impact / Severity

  • Blocker
  • High
  • Medium
  • Low

Severity is Medium rather than Low because a nondeterministic failure in a required check blocks
unrelated pull requests and invites the habit of re-running until green, which erodes the signal the
gate exists to provide.

Source

From: docs/features/potential/2026-09-07-dfdeedle-com-tests-static-seam-race-under-parallel-load.md

Activity

  1. drmoisan commented on Sep 7, 2026

    @drmoisan
    OwnerAuthor

    Correction: the static-seam race hypothesis is refuted; this is a production defect

    This issue was filed with test-isolation as the probable mechanism. A second, independent
    investigation refuted that and identified the actual cause. The title and the Summary above are
    misleading and should be re-scoped.

    Why it is not static-seam pollution

    • All three static seams DfDeedle_COM_Tests writes (MessageBoxInvoker, TableEtlInvoker,
      StoreTableEtlInvoker) are saved and restored in finally blocks, at lines 210, 235, 280,
      333, 413, 781 and 846. The class header claim at line 31 holds.
    • None of them is reachable from the failing path. GetEmailDataInViewAsync calls table.EtlAsync
      directly; TableEtlInvoker is read only by the synchronous GetEmailDataInView
      (UtilitiesCS/Extensions/DfDeedle.cs:94).
    • The sibling change to that test file replaced two reflection calls with direct calls and added no
      static mutation.

    Actual mechanism

    OlTableExtensions.EtlAsync sets milliseconds = 250 * rowCount, which is 250 ms for this one-row
    test. EtlByRowAsync awaits two Task.Run(...).TimeoutAfter(250, 3).

    The retry count is inert. UtilitiesCS/Threading/TimeOutTask.cs:824 wraps task.TimeoutAfter(ms) in
    a try/catch (TimeoutException), but that overload returns a faulted proxy Task rather than
    throwing synchronously, so the catch never fires and no retry occurs.

    On expiry, the catch (TimeoutException) in EtlAsync swallows the exception and returns
    (data!, columnDictionary) with data still null — a null passed through a null-forgiving
    suppression into a non-nullable tuple element. DfDeedle.cs:186 then dereferences
    tableSnapshot.Item1.GetLength(0).

    Timing evidence

    The same test measured 99 ms on the green run on main, 96 ms on the green re-run, and
    389 ms on the failing run, against the 250 ms deadline. The failure is a timing race against that
    deadline, not an ordering artefact.

    Why this is more than a flaky test

    A slow Outlook ETL silently yields null and surfaces as a NullReferenceException naming neither the
    folder nor the step that failed. That is the same defect class issue 798 was chartered to fix, on the
    adjacent ETL path it did not cover: the guard 798 added at DfDeedle.cs:195 runs after the
    line-186 dereference.

    Proposed fix sites, preferred first

    1. UtilitiesCS/OutlookObjects/Table/OlTableExtensions.Etl.cs — stop swallowing the
      TimeoutException into a null return. Rethrow naming the folder and the step, or return a
      nullable tuple so the caller must handle it.
    2. UtilitiesCS/Threading/TimeOutTask.cs — correct the inert retry so the wrapped overload observes
      the faulted proxy task.

    Provenance

    Diagnosed during the bugs-2026-09-06 parallel run while investigating the intermittent failure on
    pull request 802. All three candidate sites sit outside that item's declared write set, so no fix was
    attempted there; the item shipped unchanged and the analysis is recorded here instead.

  2. drmoisan commented on Sep 8, 2026

    @drmoisan
    OwnerAuthor

    Consolidated into #811 on 2026-09-07 (maintainer decision: batch review residuals by blast radius). All acceptance criteria from this issue are carried in #811; this issue is closed as superseded, not as fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions