Skip to content

test: add conformance suite and semver check - #60

Merged
wavekat-eason merged 1 commit into
mainfrom
test/backend-conformance
Aug 30, 2026
Merged

wavekat-eason merged 1 commit into
mainfrom
test/backend-conformance

Conversation

@wavekat-eason

Copy link
Copy Markdown
Contributor

Adds the two things missing from the crate's confidence story, per the gaps found while reviewing #55: a public API surface check, and one shared contract that every backend is held to.

tests/conformance.rs

Every backend implements VoiceActivityDetector, and the README tells callers they can swap one for another. That promise rests on behaviour the trait cannot enforce at compile time. Until now each backend was tested only in its own module, in its own style, and to very different depths — earshot had 15 tests, webrtc 6 — so nothing would notice a backend quietly diverging on error types, reset semantics, or what timings() counts.

This runs one identical set of checks against all five:

Check Asserts
capabilities_are_self_consistent frame_duration_ms matches frame_size at sample_rate; capabilities do not drift as audio is processed
the_declared_frame_is_accepted the backend accepts the frame size it advertises, scoring finite values in 0.0..=1.0
a_wrong_sample_rate_is_rejected InvalidSampleRate, echoing the offending rate
an_impossible_frame_length_is_rejected InvalidFrameSize with the true got length and a usable expected
reset_restores_a_fresh_detector after dirtying state, reset() reproduces a freshly constructed detector's score sequence exactly
a_rejected_call_does_not_disturb_the_stream invalid calls interleaved between valid ones do not change any score
the_frame_adapter_preserves_scores four awkward chunk sizes through FrameAdapter reproduce frame-aligned scores, with the carry always under one frame
timings_count_scored_frames steady-state frame count tracks calls; rejected calls are not counted; reset() does not clear timings

WebRTC is swept across all four sample rates and all three frame durations, and Silero across both of its rates.

Fitting the contract to all five backends

The checks are phrased to fit every backend rather than the strictest one — two cases forced that, and both are worth recording:

  • WebRTC accepts 10, 20 and 30 ms frames, not just the one it advertises in capabilities(). So the contract asserts that the declared frame size is accepted and that an impossible length is refused, rather than that only one length works.
  • FireRedVAD takes 160-sample calls but needs 400 samples for its first frame, so its opening two calls are buffering-only and score nothing. My first version of the timing check asserted frames == calls and failed here — the backend was right and the check was wrong. ProcessTimings::frames is documented as "the number of frames that produced a result (excludes buffering-only frames)", so the check is now expressed in the steady state: once warm, each further call adds exactly one frame.

All five backends pass.

semver CI job

release-plz.toml carries only tag and release settings, and there was no API check in CI. The "✓ API compatible changes" note on a release PR comes from release-plz's own check at release time — real, but not a PR gate. Nothing would have failed a PR that made a public type private or changed a signature.

The new job runs cargo-semver-checks with all-features, so the feature-gated backend modules are covered too, against the published release. make semver runs the same thing locally.

It is not decoration: I confirmed it fails as intended by removing pub from FrameAdapter::process_latest, which reports inherent_method_missing and semver requires new major version. Locally the run takes ~26s with all features, ~11s with defaults.

Verification

cargo fmt --all --check and cargo clippy --all-targets -- -D warnings clean on both the workspace and the full feature set. Tests pass across all eight feature combinations CI builds: empty, each backend alone, serde, and the full combination.

🤖 Generated with Claude Code

https://claude.ai/code/session_012PF9YKsA6GfchGTPDAWYzL

Adds the two things missing from the crate's confidence story: a public
API surface check, and one shared contract that every backend is held to.

`tests/conformance.rs` runs an identical set of checks against all five
backends: capabilities are self-consistent and stable, the declared frame
is accepted with a score in range, a wrong sample rate and an impossible
frame length are rejected with the documented error, `reset()` restores a
freshly constructed detector, a rejected call leaves streaming state
untouched, the FrameAdapter preserves scores across chunk boundaries, and
`timings()` tracks scored frames.

The checks are phrased to fit every backend rather than the strictest
one. WebRTC accepts 10/20/30 ms frames, so the contract asserts that the
declared frame size works and that an impossible length is refused, not
that only one length is accepted. FireRedVAD takes 160-sample calls but
needs 400 samples before its first frame, so the timing contract is
expressed in the steady state — which is what the trait already documents
`frames` to mean.

WebRTC is swept across all four sample rates and all three frame
durations, and Silero across both of its rates.

A `semver` CI job runs cargo-semver-checks with all features against the
published release, so an accidental break in the public API fails the PR
rather than being noticed after publishing. `make semver` runs it
locally. Verified it fails as intended by removing a public method.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012PF9YKsA6GfchGTPDAWYzL
@wavekat-eason
wavekat-eason merged commit 2dd4749 into main Aug 30, 2026
12 checks passed
@wavekat-eason
wavekat-eason deleted the test/backend-conformance branch August 30, 2026 01:11
@github-actions github-actions Bot mentioned this pull request Aug 30, 2026
wavekat-eason added a commit that referenced this pull request Aug 30, 2026
## 🤖 New release

* `wavekat-vad`: 0.1.16 -> 0.1.17 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

##
[0.1.17](v0.1.16...v0.1.17)
- 2026-08-30

### Added

- add pure-Rust earshot VAD backend
([#55](#55))

### Other

- add earshot section to the README
([#59](#59))
- add conformance suite and semver check
([#60](#60))
- cover earshot backend and frame adapter
([#57](#57))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eason WaveKat <eason@wavekat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant