Repository navigation
fix(alarms): give each audio cue its own thread instead of the pool - #335
Merged
Merged
Conversation
SerialAudioQueueTests.Enqueue_returns_without_waiting_for_playback_to_finish and A_stuck_action_does_not_permanently_block_later_ones were flaky on the Windows CI leg. They turned out to be a true positive: the fragility was in SerialAudioQueue, not in the tests. The queue drained on its own dedicated thread but ran each cue via Task.Run(play).Wait(perItemTimeout), so every cue needed a *thread-pool* thread. A pool saturated by other work hands out threads only as fast as it grows them (a thread or two per second), so a cue could sit unplayed for seconds -- or outlast its own watchdog without ever having started. Alarms fire exactly when the app is busiest, which is precisely when that bites, and an alarm that does not sound is the one failure this class exists to prevent. Run each cue on a dedicated background thread and Join it with the same bounded timeout instead. The watchdog semantics are unchanged -- a hung player is abandoned and the next cue served -- but nothing now queues behind unrelated work. Cues are rare and short, so a thread each is well affordable. Evidence. The flake would not reproduce by load alone (0 failures in 52 runs across three configurations, including the full suite under 16-way contention), so the mechanism was reproduced by construction instead: saturate the pool, then run the scenarios. Against a saturated pool, before and after this change: Old test shape, Enqueue starts playback FAIL (1s timeout) -> PASS (7 ms) Old test shape, stuck item blocks later FAIL (5s timeout) -> PASS (212 ms) Both *original* test shapes pass against the fixed queue, which is what identifies this as a production bug rather than a test bug. The tests are left alone apart from holding their long-running actions on a gate released in a finally, rather than sleeping a fixed 2s/30s, so an abandoned cue thread goes away with the test instead of idling on afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R9oidCyJ6YmhtCxu9WxGKo Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975
force-pushed
the
test/serial-audio-queue-flakes
branch
from
September 15, 2026 21:15
b982b82 to
246ce29
Compare
DeepDiver1975
added a commit
that referenced
this pull request
Sep 15, 2026
Every pull request inserted its bullet at the top of the same list under `## [Unreleased]`, so the insertion anchor was byte-identical across branches. Git saw two independent inserts at one position, could not order them, and conflicted — and merging any one PR invalidated every other open PR's hunk. `git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and nothing else, while two PRs writing to different sections merged clean. Each change now carries its entry as its own file in changelog.d/, which towncrier folds into CHANGELOG.md when a release is cut. Different PRs create different files, so there are no shared lines left to conflict over. Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the default <issue>.<type>.md: several entries cite the same ticket (#302), and issue-numbered filenames would have reintroduced the same collision as an add/add conflict. A `merge=union` driver in .gitattributes was tried first and rejected. It does resolve these merges correctly on the client, but GitHub does not run merge drivers server-side when computing pull request mergeability, so the PRs would still have shown as conflicting. The changelog-check job now runs `towncrier check` instead of requiring a CHANGELOG.md edit. Its job id and display name are unchanged, so an existing required status check keeps matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975
added a commit
that referenced
this pull request
Sep 16, 2026
Every pull request inserted its bullet at the top of the same list under `## [Unreleased]`, so the insertion anchor was byte-identical across branches. Git saw two independent inserts at one position, could not order them, and conflicted — and merging any one PR invalidated every other open PR's hunk. `git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and nothing else, while two PRs writing to different sections merged clean. Each change now carries its entry as its own file in changelog.d/, which towncrier folds into CHANGELOG.md when a release is cut. Different PRs create different files, so there are no shared lines left to conflict over. Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the default <issue>.<type>.md: several entries cite the same ticket (#302), and issue-numbered filenames would have reintroduced the same collision as an add/add conflict. A `merge=union` driver in .gitattributes was tried first and rejected. It does resolve these merges correctly on the client, but GitHub does not run merge drivers server-side when computing pull request mergeability, so the PRs would still have shown as conflicting. The changelog-check job now runs `towncrier check` instead of requiring a CHANGELOG.md edit. Its job id and display name are unchanged, so an existing required status check keeps matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975
added a commit
that referenced
this pull request
Sep 16, 2026
Every pull request inserted its bullet at the top of the same list under `## [Unreleased]`, so the insertion anchor was byte-identical across branches. Git saw two independent inserts at one position, could not order them, and conflicted — and merging any one PR invalidated every other open PR's hunk. `git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and nothing else, while two PRs writing to different sections merged clean. Each change now carries its entry as its own file in changelog.d/, which towncrier folds into CHANGELOG.md when a release is cut. Different PRs create different files, so there are no shared lines left to conflict over. Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the default <issue>.<type>.md: several entries cite the same ticket (#302), and issue-numbered filenames would have reintroduced the same collision as an add/add conflict. A `merge=union` driver in .gitattributes was tried first and rejected. It does resolve these merges correctly on the client, but GitHub does not run merge drivers server-side when computing pull request mergeability, so the PRs would still have shown as conflicting. The changelog-check job now runs `towncrier check` instead of requiring a CHANGELOG.md edit. Its job id and display name are unchanged, so an existing required status check keeps matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975
added a commit
that referenced
this pull request
Sep 20, 2026
Every pull request inserted its bullet at the top of the same list under `## [Unreleased]`, so the insertion anchor was byte-identical across branches. Git saw two independent inserts at one position, could not order them, and conflicted — and merging any one PR invalidated every other open PR's hunk. `git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and nothing else, while two PRs writing to different sections merged clean. Each change now carries its entry as its own file in changelog.d/, which towncrier folds into CHANGELOG.md when a release is cut. Different PRs create different files, so there are no shared lines left to conflict over. Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the default <issue>.<type>.md: several entries cite the same ticket (#302), and issue-numbered filenames would have reintroduced the same collision as an add/add conflict. A `merge=union` driver in .gitattributes was tried first and rejected. It does resolve these merges correctly on the client, but GitHub does not run merge drivers server-side when computing pull request mergeability, so the PRs would still have shown as conflicting. The changelog-check job now runs `towncrier check` instead of requiring a CHANGELOG.md edit. Its job id and display name are unchanged, so an existing required status check keeps matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
DeepDiver1975
added a commit
that referenced
this pull request
Sep 20, 2026
* build(changelog): assemble CHANGELOG.md from per-PR fragments Every pull request inserted its bullet at the top of the same list under `## [Unreleased]`, so the insertion anchor was byte-identical across branches. Git saw two independent inserts at one position, could not order them, and conflicted — and merging any one PR invalidated every other open PR's hunk. `git merge-tree` reproduces it: #334 and #335 conflicted on CHANGELOG.md and nothing else, while two PRs writing to different sections merged clean. Each change now carries its entry as its own file in changelog.d/, which towncrier folds into CHANGELOG.md when a release is cut. Different PRs create different files, so there are no shared lines left to conflict over. Fragments use towncrier's orphan naming (+<slug>.<type>.md) rather than the default <issue>.<type>.md: several entries cite the same ticket (#302), and issue-numbered filenames would have reintroduced the same collision as an add/add conflict. A `merge=union` driver in .gitattributes was tried first and rejected. It does resolve these merges correctly on the client, but GitHub does not run merge drivers server-side when computing pull request mergeability, so the PRs would still have shown as conflicting. The changelog-check job now runs `towncrier check` instead of requiring a CHANGELOG.md edit. Its job id and display name are unchanged, so an existing required status check keeps matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com> * docs(releasing): fold the changelog with towncrier, not by hand The release runbook still told the maintainer to "move the [Unreleased] entries under a new version heading before tagging" — the one step this branch removes. Following it after this merges produces no section at all, and with the guard from #369 in place the tag is refused outright. Replace that with the towncrier build, keeping the three things it does not do for you: the summary paragraph, the link definitions at the bottom of CHANGELOG.md, and the milestone. The tag command moves below the assembly, which is the order it now has to happen in. CONTRIBUTING.md's Hacktoberfest bullet still listed "a CHANGELOG.md entry" among the unchanged rules, contradicting the section below it that says never to edit that file; and its "Cutting a release" section now duplicated the runbook, so it points at docs/releasing.md instead of maintaining the same steps twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com> --------- Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SerialAudioQueueTests.Enqueue_returns_without_waiting_for_playback_to_finishandA_stuck_action_does_not_permanently_block_later_oneswere flaky on the Windows CI leg (seen on #327).They turned out to be a true positive. The fragility is in
SerialAudioQueue, not in the tests.Root cause
The queue drains on its own dedicated thread, but ran each cue as:
So every cue needed a thread-pool thread. A pool saturated by other work hands out threads only as fast as it grows them — a thread or two per second — so a cue could sit unplayed for seconds, or outlast its own hung-player watchdog without ever having started.
Alarms fire exactly when the app is busiest, which is precisely when this bites. An alarm that doesn't sound is the one failure this class exists to prevent.
Fix
Run each cue on a dedicated background thread and
Joinit with the same bounded timeout. Watchdog semantics are unchanged — a hung player is abandoned and the next cue served — but nothing queues behind unrelated work any more. Cues are rare and short, so a thread each is well affordable.Evidence
The flake would not reproduce by load alone — 0 failures in 52 runs across three configurations:
DOTNET_PROCESSOR_COUNT=1So rather than fix on a theory, I reproduced the mechanism by construction: saturate the thread pool, then run the scenarios. Against a saturated pool, before and after this change:
Enqueuestarts playbackEnqueuestarts playbackThe decisive row is that the original, unmodified test shapes pass against the fixed queue. That is what identifies this as a production bug rather than a test bug — and it's why I abandoned the test-rewrite I'd initially planned: a test-side change could not fix it, because under a starved pool the queue genuinely could not run anything.
Full suite green: 1,253 passed, 1 skipped.
Test changes, deliberately minimal
The tests are left as they were apart from holding their long-running actions on a gate released in a
finally, instead of sleeping a fixed 2s/30s — so an abandoned cue thread goes away with the test rather than idling on afterwards. The stale comment claiming the 5s budget was about waiting for a pool thread is gone, since that is no longer how it works.🤖 Generated with Claude Code
https://claude.ai/code/session_01R9oidCyJ6YmhtCxu9WxGKo