Repository navigation
ci: include earshot in release benchmarks - #58
Merged
Merged
Conversation
The `update-benchmarks` job in release-plz.yml is what stamps the README benchmark table onto the release PR, and its feature list still omitted `earshot`. #57 updated the list in ci.yml and vad-accuracy.yml but missed this one, so the v0.1.17 release PR carries a table with no Earshot row. Also fixes the manual VAD Accuracy workflow, whose final step could not open its PR: actions/checkout persists an Authorization header that create-pull-request duplicates, and the push fails with HTTP 400. The checkout no longer persists credentials, and the action is updated to the current major. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012PF9YKsA6GfchGTPDAWYzL
wavekat-eason
added a commit
that referenced
this pull request
Aug 30, 2026
Follow-up to #55 / #57 / #58. ## What was missing #55 added Earshot to the backends table, the dependency snippet, and the feature-flag table — but not to the part of the README where each backend is actually explained. WebRTC, Silero, TEN-VAD and FireRedVAD each have a `###` section with a description and a runnable example; Earshot went straight from the FireRedVAD example to the trait docs. Someone who enabled the feature had nowhere to look for how to construct it. The new section follows the same shape as the other four and calls out the three things that differ from them: - construction is infallible, so there is no `.unwrap()` on `new()` - the frame size is exactly 256 samples, with a pointer to `FrameAdapter` for 20 ms / 320-sample transports - each detector holds ~8 KiB of streaming state, so one per stream, and `reset()` between streams ## `FrameAdapter::process_each` The adapter rework in #55 added `process_each` and made the allocation behaviour of each method meaningful, but the README still listed only the three older methods. Added it to the example, with a note on which methods allocate and what `buffered_samples()` reports — this is the method to reach for on a real-time path, and it was invisible. ## Acknowledgements Added the earshot crate (MIT OR Apache-2.0, by pyke — the same authors as `ort`, already credited). ## Notes - Both new code examples were compiled and run against the crate. The README is not wired into doctests, so CI would not have caught a mistake in them. - I did not repeat upstream's accuracy claim. Their README states Earshot is more accurate than Silero VAD v6 and TEN VAD; on the TEN-VAD testset this crate benchmarks against, our measurement puts it at F1 0.928 — level with TEN-VAD and below Silero's 0.938, with the speed advantage intact. The benchmark table speaks for itself, so the section sticks to characteristics rather than comparisons. - The benchmark table itself is untouched. It is generated by the `update-benchmarks` job and #56 already has the regenerated five-row version, so hand-editing it here would only conflict. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012PF9YKsA6GfchGTPDAWYzL Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #57.
The release PR's table has no Earshot row
There are four places that spell out the backend feature list. #57 updated three of them, but missed the one that actually matters for a release: the
update-benchmarksjob inrelease-plz.yml. That job checks out the release PR branch, runs the accuracy report, and pushes the regenerated table onto it — and it was still running with--features webrtc,silero,ten-vad,firered.That is why #56 currently carries a table stamped v0.1.17 listing four backends, with Earshot absent from the release that introduces it.
Verified there are no other stragglers:
The manual accuracy workflow cannot open its PR
Dispatching VAD Accuracy on main measured everything correctly and updated the README in the workspace, then failed on the last step:
actions/checkoutleaves a persistedhttp.https://github.com/.extraheadercredential behind, andcreate-pull-requestadds its own — twoAuthorizationheaders, and the push is rejected. The checkout in that workflow no longer persists credentials, and the action moves from v6 to the current major (v8 is v7 plus a Node 24 runner requirement; nothing this workflow uses changed).release-plz.yml's benchmark job pushes with plaingit push, so it was never affected by this — which is why the table on the release branch updates while the manual workflow fails.Measured numbers
From the run on
ubuntu-latest, which matches the local measurement exactly on the deterministic metrics:Once this lands, re-running Release-plz regenerates the table on #56 with the Earshot row, and v0.1.17 can go out.
🤖 Generated with Claude Code
https://claude.ai/code/session_012PF9YKsA6GfchGTPDAWYzL