Skip to content

fix(alarms): give each audio cue its own thread instead of the pool - #335

Merged
DeepDiver1975 merged 1 commit into
mainfrom
test/serial-audio-queue-flakes
Sep 15, 2026
Merged

DeepDiver1975 merged 1 commit into
mainfrom
test/serial-audio-queue-flakes

Conversation

@DeepDiver1975

Copy link
Copy Markdown
Contributor

SerialAudioQueueTests.Enqueue_returns_without_waiting_for_playback_to_finish and A_stuck_action_does_not_permanently_block_later_ones were flaky on the Windows CI leg (seen on #327).

They turned out to be a true positive. The fragility is in SerialAudioQueue, not in the tests.

Root cause

The queue drains on its own dedicated thread, but ran each cue as:

Task.Run(play).Wait(_perItemTimeout);

So every cue needed a thread-pool thread. A pool saturated by other work hands out threads only as fast as it grows them — a thread or two per second — so a cue could sit unplayed for seconds, or outlast its own hung-player watchdog without ever having started.

Alarms fire exactly when the app is busiest, which is precisely when this bites. An alarm that doesn't sound is the one failure this class exists to prevent.

Fix

Run each cue on a dedicated background thread and Join it with the same bounded timeout. Watchdog semantics are unchanged — a hung player is abandoned and the next cue served — but nothing queues behind unrelated work any more. Cues are rare and short, so a thread each is well affordable.

Evidence

The flake would not reproduce by load alone — 0 failures in 52 runs across three configurations:

attempt runs failures
tests only, 12-way CPU load 30 0
tests only, DOTNET_PROCESSOR_COUNT=1 12 0
full suite, 16-way CPU load 10 0

So rather than fix on a theory, I reproduced the mechanism by construction: saturate the thread pool, then run the scenarios. Against a saturated pool, before and after this change:

scenario before after
Old test shape, Enqueue starts playback FAIL (1s timeout) PASS (7 ms)
Old test shape, stuck item blocks later FAIL (5s timeout) PASS (212 ms)
New test shape, Enqueue starts playback FAIL (5s timeout) PASS (24 ms)
New test shape, stuck item blocks later FAIL (5s timeout) PASS (208 ms)

The decisive row is that the original, unmodified test shapes pass against the fixed queue. That is what identifies this as a production bug rather than a test bug — and it's why I abandoned the test-rewrite I'd initially planned: a test-side change could not fix it, because under a starved pool the queue genuinely could not run anything.

Full suite green: 1,253 passed, 1 skipped.

Test changes, deliberately minimal

The tests are left as they were apart from holding their long-running actions on a gate released in a finally, instead of sleeping a fixed 2s/30s — so an abandoned cue thread goes away with the test rather than idling on afterwards. The stale comment claiming the 5s budget was about waiting for a pool thread is gone, since that is no longer how it works.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R9oidCyJ6YmhtCxu9WxGKo

SerialAudioQueueTests.Enqueue_returns_without_waiting_for_playback_to_finish and
A_stuck_action_does_not_permanently_block_later_ones were flaky on the Windows CI
leg. They turned out to be a true positive: the fragility was in SerialAudioQueue,
not in the tests.

The queue drained on its own dedicated thread but ran each cue via
Task.Run(play).Wait(perItemTimeout), so every cue needed a *thread-pool* thread. A
pool saturated by other work hands out threads only as fast as it grows them (a
thread or two per second), so a cue could sit unplayed for seconds -- or outlast its
own watchdog without ever having started. Alarms fire exactly when the app is
busiest, which is precisely when that bites, and an alarm that does not sound is the
one failure this class exists to prevent.

Run each cue on a dedicated background thread and Join it with the same bounded
timeout instead. The watchdog semantics are unchanged -- a hung player is abandoned
and the next cue served -- but nothing now queues behind unrelated work. Cues are
rare and short, so a thread each is well affordable.

Evidence. The flake would not reproduce by load alone (0 failures in 52 runs across
three configurations, including the full suite under 16-way contention), so the
mechanism was reproduced by construction instead: saturate the pool, then run the
scenarios. Against a saturated pool, before and after this change:

  Old test shape, Enqueue starts playback   FAIL (1s timeout)  -> PASS (7 ms)
  Old test shape, stuck item blocks later   FAIL (5s timeout)  -> PASS (212 ms)

Both *original* test shapes pass against the fixed queue, which is what identifies
this as a production bug rather than a test bug.

The tests are left alone apart from holding their long-running actions on a gate
released in a finally, rather than sleeping a fixed 2s/30s, so an abandoned cue
thread goes away with the test instead of idling on afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R9oidCyJ6YmhtCxu9WxGKo
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
@DeepDiver1975
DeepDiver1975 force-pushed the test/serial-audio-queue-flakes branch from b982b82 to 246ce29 Compare September 15, 2026 21:15
@DeepDiver1975
DeepDiver1975 merged commit 8fb4724 into main Sep 15, 2026
3 checks passed
@DeepDiver1975
DeepDiver1975 deleted the test/serial-audio-queue-flakes branch September 15, 2026 22:49
DeepDiver1975 added a commit that referenced this pull request Sep 15, 2026
Every pull request inserted its bullet at the top of the same list under
`## [Unreleased]`, so the insertion anchor was byte-identical across branches.
Git saw two independent inserts at one position, could not order them, and
conflicted — and merging any one PR invalidated every other open PR's hunk.
`git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and
nothing else, while two PRs writing to different sections merged clean.

Each change now carries its entry as its own file in changelog.d/, which
towncrier folds into CHANGELOG.md when a release is cut. Different PRs create
different files, so there are no shared lines left to conflict over.

Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the
default <issue>.<type>.md: several entries cite the same ticket (#302), and
issue-numbered filenames would have reintroduced the same collision as an
add/add conflict.

A `merge=union` driver in .gitattributes was tried first and rejected. It does
resolve these merges correctly on the client, but GitHub does not run merge
drivers server-side when computing pull request mergeability, so the PRs would
still have shown as conflicting.

The changelog-check job now runs `towncrier check` instead of requiring a
CHANGELOG.md edit. Its job id and display name are unchanged, so an existing
required status check keeps matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975 added a commit that referenced this pull request Sep 16, 2026
Every pull request inserted its bullet at the top of the same list under
`## [Unreleased]`, so the insertion anchor was byte-identical across branches.
Git saw two independent inserts at one position, could not order them, and
conflicted — and merging any one PR invalidated every other open PR's hunk.
`git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and
nothing else, while two PRs writing to different sections merged clean.

Each change now carries its entry as its own file in changelog.d/, which
towncrier folds into CHANGELOG.md when a release is cut. Different PRs create
different files, so there are no shared lines left to conflict over.

Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the
default <issue>.<type>.md: several entries cite the same ticket (#302), and
issue-numbered filenames would have reintroduced the same collision as an
add/add conflict.

A `merge=union` driver in .gitattributes was tried first and rejected. It does
resolve these merges correctly on the client, but GitHub does not run merge
drivers server-side when computing pull request mergeability, so the PRs would
still have shown as conflicting.

The changelog-check job now runs `towncrier check` instead of requiring a
CHANGELOG.md edit. Its job id and display name are unchanged, so an existing
required status check keeps matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975 added a commit that referenced this pull request Sep 16, 2026
Every pull request inserted its bullet at the top of the same list under
`## [Unreleased]`, so the insertion anchor was byte-identical across branches.
Git saw two independent inserts at one position, could not order them, and
conflicted — and merging any one PR invalidated every other open PR's hunk.
`git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and
nothing else, while two PRs writing to different sections merged clean.

Each change now carries its entry as its own file in changelog.d/, which
towncrier folds into CHANGELOG.md when a release is cut. Different PRs create
different files, so there are no shared lines left to conflict over.

Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the
default <issue>.<type>.md: several entries cite the same ticket (#302), and
issue-numbered filenames would have reintroduced the same collision as an
add/add conflict.

A `merge=union` driver in .gitattributes was tried first and rejected. It does
resolve these merges correctly on the client, but GitHub does not run merge
drivers server-side when computing pull request mergeability, so the PRs would
still have shown as conflicting.

The changelog-check job now runs `towncrier check` instead of requiring a
CHANGELOG.md edit. Its job id and display name are unchanged, so an existing
required status check keeps matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975 added a commit that referenced this pull request Sep 20, 2026
Every pull request inserted its bullet at the top of the same list under
`## [Unreleased]`, so the insertion anchor was byte-identical across branches.
Git saw two independent inserts at one position, could not order them, and
conflicted — and merging any one PR invalidated every other open PR's hunk.
`git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and
nothing else, while two PRs writing to different sections merged clean.

Each change now carries its entry as its own file in changelog.d/, which
towncrier folds into CHANGELOG.md when a release is cut. Different PRs create
different files, so there are no shared lines left to conflict over.

Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the
default <issue>.<type>.md: several entries cite the same ticket (#302), and
issue-numbered filenames would have reintroduced the same collision as an
add/add conflict.

A `merge=union` driver in .gitattributes was tried first and rejected. It does
resolve these merges correctly on the client, but GitHub does not run merge
drivers server-side when computing pull request mergeability, so the PRs would
still have shown as conflicting.

The changelog-check job now runs `towncrier check` instead of requiring a
CHANGELOG.md edit. Its job id and display name are unchanged, so an existing
required status check keeps matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975 added a commit that referenced this pull request Sep 20, 2026
* build(changelog): assemble CHANGELOG.md from per-PR fragments

Every pull request inserted its bullet at the top of the same list under
`## [Unreleased]`, so the insertion anchor was byte-identical across branches.
Git saw two independent inserts at one position, could not order them, and
conflicted — and merging any one PR invalidated every other open PR's hunk.
`git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and
nothing else, while two PRs writing to different sections merged clean.

Each change now carries its entry as its own file in changelog.d/, which
towncrier folds into CHANGELOG.md when a release is cut. Different PRs create
different files, so there are no shared lines left to conflict over.

Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the
default <issue>.<type>.md: several entries cite the same ticket (#302), and
issue-numbered filenames would have reintroduced the same collision as an
add/add conflict.

A `merge=union` driver in .gitattributes was tried first and rejected. It does
resolve these merges correctly on the client, but GitHub does not run merge
drivers server-side when computing pull request mergeability, so the PRs would
still have shown as conflicting.

The changelog-check job now runs `towncrier check` instead of requiring a
CHANGELOG.md edit. Its job id and display name are unchanged, so an existing
required status check keeps matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>

* docs(releasing): fold the changelog with towncrier, not by hand

The release runbook still told the maintainer to "move the [Unreleased]
entries under a new version heading before tagging" — the one step this
branch removes. Following it after this merges produces no section at all,
and with the guard from #369 in place the tag is refused outright.

Replace that with the towncrier build, keeping the three things it does not
do for you: the summary paragraph, the link definitions at the bottom of
CHANGELOG.md, and the milestone. The tag command moves below the assembly,
which is the order it now has to happen in.

CONTRIBUTING.md's Hacktoberfest bullet still listed "a CHANGELOG.md entry"
among the unchanged rules, contradicting the section below it that says
never to edit that file; and its "Cutting a release" section now duplicated
the runbook, so it points at docs/releasing.md instead of maintaining the
same steps twice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>

---------

Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant