Skip to content

docs: add SLO-208 benchmark signal split spec - #63

Merged
messagesgoel-blip merged 3 commits into
mainfrom
feat/slo-208-spec
May 17, 2026
Merged

messagesgoel-blip merged 3 commits into
mainfrom
feat/slo-208-spec

Conversation

@messagesgoel-blip

@messagesgoel-blip messagesgoel-blip commented May 17, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #62

Project: Slopgate Roadmap

Summary

  • add the SLO-208 benchmark signal split and Whimsy gap-closure spec to docs
  • capture the benchmark scoring split, noise-reduction work, and first semantic gap-closure pass
  • anchor the implementation plan in the repo before breaking out sub-issues

Validation

  • Documented benchmark corpus archive, source commit, invocation, and Whimsy base/head SHAs
  • Added deterministic severity rubrics for dual-severity semantic rule concepts
  • Made the SLP068 completion threshold measurable at at least 70% reduction

Review

  • CodeRabbit comments addressed
  • CodeRabbit approved latest commit

@coderabbitai

coderabbitai Bot commented May 17, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Rate limit exceeded

@messagesgoel-blip has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 1 minute and 38 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: f4125181-54fc-4c68-866b-fa9fd92cb1af

📥 Commits

Reviewing files that changed from the base of the PR and between 20ea482 and 72271c7.

📒 Files selected for processing (1)
  • docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
📝 Walkthrough

Walkthrough

Adds a new draft specification that defines a tiered benchmark scoring model, targeted noise-reduction changes for seven SLP rules, a first wave of semantic gap-closure rules, and a phased implementation, validation, and acceptance plan anchored to the Whimsy benchmark corpus.

Changes

SLO-208 Benchmark Signal Split and Whimsy Gap Closure

Layer / File(s) Summary
Spec metadata and title
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Spec title, date, status, roadmap task, tracking issue, target repo, and benchmark window (lines 1–12).
Benchmark corpus anchor and reproducibility
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Baseline toolchain/generator settings, reproducible invocation template, archived benchmark artifacts, and Whimsy corpus SHAs (lines 13–67).
Problem statement and noise quantification
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Observed benchmark output composition, low-signal coverage/precision proxies, Whimsy dominance, and archive truncation behavior (lines 68–104).
Goals, non-goals, and scope
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Goals and non-goals, worst pollution examples, and overall scope/deliverables for scoring, noise reduction, and semantic gap-closure (lines 105–147).
Tiered scoring and eligibility design
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Tiered scoring groups; per-finding/per-rule eligibility modes (parity/advisory/disabled); expected advisory handling (lines 148–172).
Rule-specific noise reduction requirements
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Concrete requirements and acceptance targets for SLP068, SLP035, SLP053, SLP017, SLP056, SLP033, and SLP081 including artifact filtering, context checks, and advisory/eligibility changes (lines 173–265).
First-wave semantic gap-closure rules
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
New rule concepts and severities: OpenAPI secret-field split, API-wrapper vs consumer response-shape drift, OpenAPI merge-order override hazard, and order-sensitive assertions (lines 267–306).
Likely touched files and placeholders
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Lists likely implementation files and placeholders for new registry/implementation and tests (lines 307–322).
Phased implementation plan
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Phased delivery plan covering scoring, rule retuning, semantic wave, benchmarking, and precision gating (lines 324–345).
Validation and benchmark re-runs
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Unit/regression test expectations and detailed benchmark re-run requirements against Whimsy corpus SHAs with tiered before/after reporting (lines 346–361).
Acceptance criteria
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Required benchmark metrics, specific rule eligibility/noise reduction outcomes, and confirmation of at least one new semantic rule with tests (lines 362–372).
Risks and open questions
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
Open questions about advisory handling, generated-artifact ignore strategy, eligibility aggressiveness, and benchmark truncation policy (lines 373–383).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

enhancement, no-roadmap-task

Poem

I hopped through specs at dawn's first light,
Sorting Whimsy's whispers from true insight,
Seven rules tuned, four new rules to try,
Benchmarks rerun beneath the sky,
🐰 — a cheerful rabbit closes the gap with a sigh.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive The PR is documentation-only and scopes the implementation plan for issue #62; coding requirements (rule retuning, metric implementation, new semantic rules) are documented as future work, not implemented in this PR. Clarify whether this spec-only PR should satisfy issue #62 coding requirements or if those are deferred to follow-up PRs. If deferred, the linked issue objectives may be incomplete for this PR to close issue #62.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding a specification document for the SLO-208 benchmark signal split feature.
Description check ✅ Passed The description follows the template with issue reference, project context, summary, validation checklist, and review status; all required sections are present and substantive.
Out of Scope Changes check ✅ Passed The PR adds only a specification document; no code changes are present. The documented deliverables and implementation plan align with issue #62 scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/slo-208-spec

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (1)
  • LINEAR integration encountered authorization issues. Please disconnect and reconnect the integration in the CodeRabbit UI.

Comment @coderabbitai help to get the list of available commands and usage tips.

@messagesgoel-blip

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented May 17, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md`:
- Around line 9-10: The spec currently names only the benchmark time window
("Primary benchmark window analyzed") but lacks a reproducibility anchor; update
the document to include a concrete benchmark corpus pointer by adding the
benchmark archive path or storage URI, the exact commit SHA(s) used to produce
the corpus, and the exact invocation (command, flags, environment variables and
tool versions) used to generate and run the benchmark so others can reproduce
the before/after numbers; make the same addition to the other referenced section
(lines 291-293) and reference these anchors (archive URI + commit SHA +
command/options) wherever "benchmark corpus" or "PR list" is discussed.
- Line 228: The spec currently allows dual severities like "block" or "warn"
without criteria; update each dual-severity rule definition to include a
deterministic severity rubric: add explicit confidence thresholds (e.g.,
confidence >= 0.9 -> block, 0.6–0.9 -> warn, <0.6 -> none) or a predicate field
(e.g., severityPredicate) that evaluates to "block" or "warn" so the resolver
(or evaluator) can deterministically pick one severity; apply this change to
every rule that lists multiple severities (the entries mentioning "block" or
"warn") and document the threshold/predicate schema next to those rule
definitions.
- Around line 305-306: Update the SLP068 acceptance text to use the explicit,
measurable threshold from the Acceptance Criteria: replace "SLP068 visible noise
drops materially on the recent Whimsy corpus" with a statement that SLP068
visible noise is reduced by ≥70% on the specified Whimsy corpus (or the exact
threshold shown in the Acceptance Criteria), so the completion criterion is
objective and testable; ensure the revised wording references "SLP068" and
"Whimsy corpus" exactly and matches the threshold wording used earlier in the
Acceptance Criteria.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: fd0f25ed-7d94-4d5d-8c9a-6691524217b4

📥 Commits

Reviewing files that changed from the base of the PR and between 1d1a16b and 0b3a354.

📒 Files selected for processing (1)
  • docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
📜 Review details
🧰 Additional context used
🪛 LanguageTool
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md

[style] ~33-~33: Consider removing “of” to be more concise
Context: ...3 / 1256 = 0.24%` Whimsy drives nearly all of the problem: - 10 of the 12 runs were Whim...

(ALL_OF_THE)


[style] ~51-~51: Specify a number, remove phrase, use “a few”, or use “some”
Context: ...e. 2. Reduce Whimsy benchmark spam from a small number of noisy rules. 3. Preserve real blocker c...

(SMALL_NUMBER_OF)


[style] ~301-~301: Consider removing “of” to be more concise
Context: ...ance Criteria The arc is complete when all of the following are true: - Benchmark output...

(ALL_OF_THE)


[style] ~320-~320: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ule logic, config defaults, or both? 3. Should benchmark truncation stay at 50 detail ...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

@messagesgoel-blip

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@messagesgoel-blip
messagesgoel-blip marked this pull request as ready for review May 17, 2026 22:49
@coderabbitai

coderabbitai Bot commented May 17, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot added enhancement New feature or request no-roadmap-task Allowed exception: no roadmap task required labels May 17, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md`:
- Around line 302-305: The three bullets about severity for exact-order
assertions miss the case where the assertion is test-only but a production
contract artifact is changed in the same diff; update the spec text around the
three bullets (the lines describing "warn", "info", "no finding" for exact-order
assertions) to explicitly cover this edge case by either (a) adding a fourth
bullet that states "warn when the exact-order assertion is test-only AND a
production contract artifact is changed in the same diff" or (b) rewording the
second bullet to read "info when the exact-order assertion is test-only and no
production contract artifact is changed in the same diff; if a production
contract artifact is changed, treat as warn", so the rules for exact-order
assertion + production artifact change are unambiguous.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: bb904ede-8e51-4ef0-8876-ec2047e51067

📥 Commits

Reviewing files that changed from the base of the PR and between 0b3a354 and 20ea482.

📒 Files selected for processing (1)
  • docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md
📜 Review details
🧰 Additional context used
🪛 LanguageTool
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md

[style] ~383-~383: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ule logic, config defaults, or both? 3. Should benchmark truncation stay at 50 detail ...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

🔇 Additional comments (6)
docs/superpowers/specs/2026-05-17-slo-208-benchmark-signal-split.md (6)

1-67: LGTM!


68-172: LGTM!


173-266: LGTM!


271-295: LGTM!


307-372: LGTM!


373-384: LGTM!

@messagesgoel-blip
messagesgoel-blip merged commit 215849f into main May 17, 2026
3 of 4 checks passed
@messagesgoel-blip
messagesgoel-blip deleted the feat/slo-208-spec branch May 25, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request no-roadmap-task Allowed exception: no roadmap task required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SLO-208 — Benchmark signal split and Whimsy gap closure

1 participant