Repository navigation
SLO-208A: add benchmark scoring tiers and eligibility metadata - #68
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
🔇 Additional comments (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR adds tiered benchmark scoring (all_rules, block_warn_only, benchmark_eligible), derives benchmark_mode from finding severity with per-rule overrides, emits per-tier results and metadata in the JSON payload, and updates compare tooling to display tier-aware columns and aggregated totals while keeping legacy top-level fields. ChangesBenchmark Scoring Tiers
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsErrors were encountered while retrieving linked issues. Errors (1)
Comment |
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/benchmark_review.py`:
- Around line 611-615: The current lookup silently treats unknown
finding["severity"] as "parity"; instead, validate severity against the known
severity set (the keys of BENCHMARK_MODE_BY_SEVERITY) before using it. In the
block that computes benchmark_mode (which currently references
RULE_BENCHMARK_MODE_OVERRIDES, finding["rule_id"], BENCHMARK_MODE_BY_SEVERITY,
and severity), if severity is not in BENCHMARK_MODE_BY_SEVERITY.keys() raise an
explicit error (e.g., ValueError or custom exception) or mark the finding as
ineligible, so unrecognized severities fail closed rather than defaulting to
"parity". Ensure the check happens prior to reading BENCHMARK_MODE_BY_SEVERITY
and that the error message includes the offending finding["rule_id"] and
severity for debugging.
In `@scripts/benchmark-compare.py`:
- Around line 89-102: get_stream_total currently indexes legacy_map with
stream_name which can raise KeyError for unexpected names; update
get_stream_total to defensively handle unknown stream_name by checking if
stream_name exists in legacy_map (or using legacy_map.get(stream_name)) before
accessing legacy, and if missing return 0 (or use a safe fallback) so that
legacy = data.get(legacy_key, {}) is only attempted with a valid legacy_key;
reference the legacy_map, stream_name, streams and total variables in the
get_stream_total function when implementing this guard.
- Around line 291-295: The helper _tier_metric_value currently returns
metrics[key] without verifying the key exists which can raise KeyError; update
it to defensively handle missing keys by using the metrics.get(key, "-") pattern
(or catching KeyError) so it returns the same "-" fallback used when
tier_metrics(...) is None; locate this change in the _tier_metric_value function
and keep the existing behavior when metrics is missing, referencing tier_metrics
and the tier_name/key parameters.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: a8a6410f-a5de-4eae-8acb-820de22bf519
📒 Files selected for processing (3)
scripts/benchmark-compare.pyscripts/benchmark_review.pyscripts/benchmark_review_test.py
📜 Review details
🧰 Additional context used
🪛 Ruff (0.15.12)
scripts/benchmark-compare.py
[warning] 291-291: Dynamically typed expressions (typing.Any) are disallowed in _tier_metric_value
(ANN401)
🔇 Additional comments (8)
scripts/benchmark_review_test.py (1)
327-426: LGTM!scripts/benchmark-compare.py (7)
29-34: LGTM!
70-86: LGTM!
105-116: LGTM!
119-128: LGTM!
131-144: LGTM!
147-197: LGTM!
237-265: LGTM!
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/benchmark_compare_test.py`:
- Line 9: Split the compound assertion "assert SPEC and SPEC.loader" into two
separate assertions to provide clearer failure messages: first assert that SPEC
is truthy (e.g., "assert SPEC, 'SPEC is not set'") and then assert that
SPEC.loader is truthy (e.g., "assert SPEC.loader, 'SPEC.loader is not set'");
update the assertion locations where SPEC and SPEC.loader are used in the script
(refer to the SPEC symbol and its loader attribute) so failures indicate which
part of the import setup failed.
- Around line 15-50: Add positive-path unit tests to BenchmarkCompareHelpersTest
to assert correct returns when data is present: add a test that calls
benchmark_compare.get_stream_total with data containing "streams":
{"coderabbit_all": {"total": 3}} (or similar) and asserts it returns 3, and add
a test that calls benchmark_compare._tier_metric_value with data containing
"benchmark_tiers": {"all_rules": {"scores": {"overlap_all": 2}}} (and matching
"streams" if needed) and asserts it returns the numeric metric (e.g., 2) instead
of "-" so the normal/valid-code paths for get_stream_total and
_tier_metric_value are covered.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7fee9f53-862b-4b6b-b551-83206994053c
📒 Files selected for processing (4)
scripts/benchmark-compare.pyscripts/benchmark_compare_test.pyscripts/benchmark_review.pyscripts/benchmark_review_test.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Seer Code Review
🧰 Additional context used
🪛 Ruff (0.15.12)
scripts/benchmark_compare_test.py
[warning] 9-9: Assertion should be broken down into multiple parts
Break down assertion into multiple parts
(PT018)
scripts/benchmark-compare.py
[warning] 296-296: Dynamically typed expressions (typing.Any) are disallowed in _tier_metric_value
(ANN401)
scripts/benchmark_review_test.py
[warning] 367-367: Use pytest.raises instead of unittest-style assertRaisesRegex
Replace assertRaisesRegex with pytest.raises
(PT027)
scripts/benchmark_review.py
[warning] 613-615: Avoid specifying long messages outside the exception class
(TRY003)
🔇 Additional comments (20)
scripts/benchmark_review_test.py (1)
327-441: LGTM!scripts/benchmark-compare.py (9)
29-34: LGTM!
70-86: LGTM!
89-107: LGTM!
110-121: LGTM!
124-133: LGTM!
136-149: LGTM!
152-199: LGTM!
242-278: LGTM!
296-300: LGTM!scripts/benchmark_review.py (10)
29-39: LGTM!
610-628: LGTM!
631-641: LGTM!
644-659: LGTM!
662-697: LGTM!
700-735: LGTM!
738-743: LGTM!
746-799: LGTM!
944-956: LGTM!
960-969: LGTM!
Closes #64
Summary