Skip to content

SLO-208A: add benchmark scoring tiers and eligibility metadata - #68

Merged
messagesgoel-blip merged 3 commits into
mainfrom
feat/slo-208a-benchmark-scoring
May 17, 2026
Merged

messagesgoel-blip merged 3 commits into
mainfrom
feat/slo-208a-benchmark-scoring

Conversation

@messagesgoel-blip

Copy link
Copy Markdown
Collaborator

Closes #64

Summary

  • add benchmark scoring tiers for all rules, block/warn only, and benchmark-eligible findings
  • preserve the legacy top-level benchmark fields while emitting the new tiered JSON payload
  • update benchmark comparison output and add regression coverage for the new payload shape

@coderabbitai

coderabbitai Bot commented May 17, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9497b659-54a0-485b-9788-0202d2aaef6f

📥 Commits

Reviewing files that changed from the base of the PR and between 4fe8ffb and 2a4e9bf.

📒 Files selected for processing (1)
  • scripts/benchmark_compare_test.py
📜 Recent review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Seer Code Review
🔇 Additional comments (1)
scripts/benchmark_compare_test.py (1)

1-13: LGTM!

Also applies to: 16-78, 80-81


📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Tier-based benchmarking and reporting added with per-tier metrics and aggregated totals for "All Rules", "Block/Warn", and "Eligible".
    • Comparison output now includes compact per-tier summaries (SG/Overlap/Coverage/Precision) and tier-aware table columns; legacy top-level fields are preserved.
    • Findings are categorized by severity and benchmark mode, with benchmark-eligibility tracking and legacy-compatible fallbacks for older report formats.
  • Tests

    • Added unit tests covering tiering logic and comparison helper behavior.

Walkthrough

This PR adds tiered benchmark scoring (all_rules, block_warn_only, benchmark_eligible), derives benchmark_mode from finding severity with per-rule overrides, emits per-tier results and metadata in the JSON payload, and updates compare tooling to display tier-aware columns and aggregated totals while keeping legacy top-level fields.

Changes

Benchmark Scoring Tiers

Layer / File(s) Summary
Tier configuration and display labels
scripts/benchmark-compare.py, scripts/benchmark_review.py
Adds TIER_ORDER/TIER_LABELS and BENCHMARK_MODE_BY_SEVERITY/RULE_BENCHMARK_MODE_OVERRIDES/BENCHMARK_TIER_DESCRIPTIONS constants for tier selection and mode mapping.
Compare helpers and table formatting
scripts/benchmark-compare.py
Adds tier-aware helpers (get_tier, get_stream_total, tier_metrics, format_tier_summary, aggregate_tier_metrics) and rewrites format_table to emit per-benchmark and aggregated tier columns ("All Rules", "Block/Warn", "Eligible").
compare_files metric wiring
scripts/benchmark-compare.py
Updates compare_files to iterate dynamic tier-driven metrics (Slopgate/Overlap/Coverage/Precision per tier) and adds _tier_metric_value for cell lookup.
Finding normalization and severity→mode mapping
scripts/benchmark_review.py
Normalizes Slopgate findings to lowercased severity, derives benchmark_mode via severity mapping and rule overrides, and sets benchmark_eligible on comparison items.
Per-tier result construction
scripts/benchmark_review.py
Implements build_tier_result and build_benchmark_tiers to match findings against review streams and produce structured per-tier summaries, stream totals, and computed score summaries.
Payload assembly and main integration
scripts/benchmark_review.py
Adds build_result_payload that preserves legacy top-level keys (sourced from the all_rules tier), emits benchmark_tiers and benchmark_metadata, refactors main() to use it, and prints per-tier score lines.
Unit tests for compare and review helpers
scripts/benchmark_compare_test.py, scripts/benchmark_review_test.py
Adds tests for get_stream_total fallback behavior and _tier_metric_value, and BenchmarkTieringTest covering finding_to_compare_item and build_result_payload legacy-compatibility and tier outputs.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

  • #62: Parent issue for benchmark tiering; this PR implements the tiered scoring infrastructure and metadata objectives described in #62.

Poem

🐇 I hopped through tiers beneath a moonlit sky,

All rules, warn, and eligible counted by,
Severities sorted, modes softly assigned,
Scores rolled up neat, legacy fields aligned,
A carrot-shaped table — benchmarks refined.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 15.63% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding benchmark scoring tiers and eligibility metadata.
Description check ✅ Passed The description is mostly complete with a clear summary of changes, though it does not fully follow the template structure with Validation and Review sections.
Linked Issues check ✅ Passed The PR successfully implements all three tiers (all_rules, block_warn_only, benchmark_eligible), preserves legacy fields, and adds regression test coverage as required by issue #64.
Out of Scope Changes check ✅ Passed All changes directly support the linked issue objectives: tier definitions, finding eligibility mapping, tiered result construction, and comprehensive test coverage.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/slo-208a-benchmark-scoring

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (1)
  • LINEAR integration encountered authorization issues. Please disconnect and reconnect the integration in the CodeRabbit UI.

Comment @coderabbitai help to get the list of available commands and usage tips.

@messagesgoel-blip

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented May 17, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/benchmark_review.py`:
- Around line 611-615: The current lookup silently treats unknown
finding["severity"] as "parity"; instead, validate severity against the known
severity set (the keys of BENCHMARK_MODE_BY_SEVERITY) before using it. In the
block that computes benchmark_mode (which currently references
RULE_BENCHMARK_MODE_OVERRIDES, finding["rule_id"], BENCHMARK_MODE_BY_SEVERITY,
and severity), if severity is not in BENCHMARK_MODE_BY_SEVERITY.keys() raise an
explicit error (e.g., ValueError or custom exception) or mark the finding as
ineligible, so unrecognized severities fail closed rather than defaulting to
"parity". Ensure the check happens prior to reading BENCHMARK_MODE_BY_SEVERITY
and that the error message includes the offending finding["rule_id"] and
severity for debugging.

In `@scripts/benchmark-compare.py`:
- Around line 89-102: get_stream_total currently indexes legacy_map with
stream_name which can raise KeyError for unexpected names; update
get_stream_total to defensively handle unknown stream_name by checking if
stream_name exists in legacy_map (or using legacy_map.get(stream_name)) before
accessing legacy, and if missing return 0 (or use a safe fallback) so that
legacy = data.get(legacy_key, {}) is only attempted with a valid legacy_key;
reference the legacy_map, stream_name, streams and total variables in the
get_stream_total function when implementing this guard.
- Around line 291-295: The helper _tier_metric_value currently returns
metrics[key] without verifying the key exists which can raise KeyError; update
it to defensively handle missing keys by using the metrics.get(key, "-") pattern
(or catching KeyError) so it returns the same "-" fallback used when
tier_metrics(...) is None; locate this change in the _tier_metric_value function
and keep the existing behavior when metrics is missing, referencing tier_metrics
and the tier_name/key parameters.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a8a6410f-a5de-4eae-8acb-820de22bf519

📥 Commits

Reviewing files that changed from the base of the PR and between 1d1a16b and d83340e.

📒 Files selected for processing (3)
  • scripts/benchmark-compare.py
  • scripts/benchmark_review.py
  • scripts/benchmark_review_test.py
📜 Review details
🧰 Additional context used
🪛 Ruff (0.15.12)
scripts/benchmark-compare.py

[warning] 291-291: Dynamically typed expressions (typing.Any) are disallowed in _tier_metric_value

(ANN401)

🔇 Additional comments (8)
scripts/benchmark_review_test.py (1)

327-426: LGTM!

scripts/benchmark-compare.py (7)

29-34: LGTM!


70-86: LGTM!


105-116: LGTM!


119-128: LGTM!


131-144: LGTM!


147-197: LGTM!


237-265: LGTM!

Comment thread scripts/benchmark_review.py Resolved
Comment thread scripts/benchmark-compare.py Resolved
Comment thread scripts/benchmark-compare.py Resolved
@messagesgoel-blip
messagesgoel-blip marked this pull request as ready for review May 17, 2026 14:18

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/benchmark_compare_test.py`:
- Line 9: Split the compound assertion "assert SPEC and SPEC.loader" into two
separate assertions to provide clearer failure messages: first assert that SPEC
is truthy (e.g., "assert SPEC, 'SPEC is not set'") and then assert that
SPEC.loader is truthy (e.g., "assert SPEC.loader, 'SPEC.loader is not set'");
update the assertion locations where SPEC and SPEC.loader are used in the script
(refer to the SPEC symbol and its loader attribute) so failures indicate which
part of the import setup failed.
- Around line 15-50: Add positive-path unit tests to BenchmarkCompareHelpersTest
to assert correct returns when data is present: add a test that calls
benchmark_compare.get_stream_total with data containing "streams":
{"coderabbit_all": {"total": 3}} (or similar) and asserts it returns 3, and add
a test that calls benchmark_compare._tier_metric_value with data containing
"benchmark_tiers": {"all_rules": {"scores": {"overlap_all": 2}}} (and matching
"streams" if needed) and asserts it returns the numeric metric (e.g., 2) instead
of "-" so the normal/valid-code paths for get_stream_total and
_tier_metric_value are covered.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7fee9f53-862b-4b6b-b551-83206994053c

📥 Commits

Reviewing files that changed from the base of the PR and between d83340e and 4fe8ffb.

📒 Files selected for processing (4)
  • scripts/benchmark-compare.py
  • scripts/benchmark_compare_test.py
  • scripts/benchmark_review.py
  • scripts/benchmark_review_test.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Seer Code Review
🧰 Additional context used
🪛 Ruff (0.15.12)
scripts/benchmark_compare_test.py

[warning] 9-9: Assertion should be broken down into multiple parts

Break down assertion into multiple parts

(PT018)

scripts/benchmark-compare.py

[warning] 296-296: Dynamically typed expressions (typing.Any) are disallowed in _tier_metric_value

(ANN401)

scripts/benchmark_review_test.py

[warning] 367-367: Use pytest.raises instead of unittest-style assertRaisesRegex

Replace assertRaisesRegex with pytest.raises

(PT027)

scripts/benchmark_review.py

[warning] 613-615: Avoid specifying long messages outside the exception class

(TRY003)

🔇 Additional comments (20)
scripts/benchmark_review_test.py (1)

327-441: LGTM!

scripts/benchmark-compare.py (9)

29-34: LGTM!


70-86: LGTM!


89-107: LGTM!


110-121: LGTM!


124-133: LGTM!


136-149: LGTM!


152-199: LGTM!


242-278: LGTM!


296-300: LGTM!

scripts/benchmark_review.py (10)

29-39: LGTM!


610-628: LGTM!


631-641: LGTM!


644-659: LGTM!


662-697: LGTM!


700-735: LGTM!


738-743: LGTM!


746-799: LGTM!


944-956: LGTM!


960-969: LGTM!

Comment thread scripts/benchmark_compare_test.py Resolved
Comment thread scripts/benchmark_compare_test.py Resolved
@messagesgoel-blip
messagesgoel-blip merged commit 19cf7d1 into main May 17, 2026
4 checks passed
@messagesgoel-blip
messagesgoel-blip deleted the feat/slo-208a-benchmark-scoring branch May 17, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SLO-208A — Benchmark scoring tiers and eligibility metadata

1 participant