Skip to content

Bug: engine-toggle-prime-marker-registration-races-removal #944

Description

@drmoisan
  • Work Mode: full-bug

Summary

In EngineToggleStateCoordinator, a prime that completes synchronously in a non-success state can run CompletePrime's marker removal before StartPrimeIfNeeded registers that marker. A stale marker for a finished prime is then left in _primeTasks, which blocks any later re-prime for that engine. This is hazard B from the #942 research, first recorded as NB-2 in the #735 code review, and never promoted.

Environment

  • OS/version: Windows 11 / windows-latest
  • Python version: n/a (C#, .NET Framework 4.8)
  • Command/flags used: n/a (production code path; also reachable from the re-prime in GetPressed_WhenPrimeFaults_LogsErrorAndStillReturnsFalse)
  • Data source or fixture: n/a

Steps to Reproduce

  1. Arrange for EngineActiveAsync to return an already-faulted task, for example after a cached configuration-load fault.
  2. Call GetPressed for that engine, which triggers StartPrimeIfNeeded.
  3. StartObservedPrime schedules the continuation. The continuation can run CompletePrime (_primeTasks.TryRemove, line ~348) before the assignment _primeTasks[engineName] = ... at line ~276 stores it.

Expected Behavior

A finished prime never leaves a marker in _primeTasks, so a later GetPressed can start a fresh prime.

Actual Behavior

The removal can precede the registration. The marker for the completed prime then stays registered, ContainsKey returns true, and no later prime starts for that engine.

Logs / Screenshots

  • Attached minimal logs or screenshot
  • Snippet: research record docs/features/active/2026-09-29-engine-toggle-prime-fault-logging-test-races-942/research/2026-09-29T23-20-engine-toggle-prime-fault-race-research.md, conclusion 3.

Impact / Severity

  • Blocker
  • High
  • Medium
  • Low

Source

From: docs/features/potential/2026-09-30-engine-toggle-prime-marker-registration-races-removal.md

Activity

  1. drmoisan commented on Sep 30, 2026

    @drmoisan
    OwnerAuthor

    Restoring sections dropped by the promotion tooling.

    Suspected Cause / Notes

    StartPrimeIfNeeded holds _primeGate while it registers, but CompletePrime removes the marker without taking the gate. Registration (line ~276) happens after the continuation is scheduled, so a continuation that finishes quickly can remove the marker (line ~348) before it is written. No current test fails on this. #942 fixes a different ordering (log before removal) and leaves this one out of scope by design. First recorded as NB-2 in the #735 code review.

    Source: docs/features/active/2026-09-29-engine-toggle-prime-fault-logging-test-races-942/research/2026-09-29T23-20-engine-toggle-prime-fault-race-research.md, conclusion 3.

    Proposed Fix / Validation Ideas

    • Write a deterministic regression test: use a pre-faulted EngineActiveAsync and assert that a second GetPressed starts a new prime. It must fail before the fix.
    • Fix options: register a placeholder before scheduling, or make removal conditional on the stored task being the completing one (TryRemove with a KeyValuePair comparison), or take _primeGate in CompletePrime.
    • Use no sleeps, retries, [DoNotParallelize] or Workers=1.
    • Sequence this after Bug: engine-toggle-prime-fault-logging-test-races #942 merges, because both change CompletePrime.
  2. drmoisan commented on Sep 30, 2026

    @drmoisan
    OwnerAuthor

    Delivered by PR 954 (merge commit 829ad41), parallel run bugs-2026-09-28. All 18 acceptance criteria in spec.md are checked off; all seven checks green on head f12b8d3 (run 36738150094) before merge. The approved one-time P3-T8 restart passed 7327 of 7327. Follow-ups are filed separately by the run coordinator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions