Repository navigation
Add Trim/NativeAOT safety rules to code review skill - #34962
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34962Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34962" |
There was a problem hiding this comment.
Pull request overview
This PR updates the .github/skills/code-review skill documentation to add explicit Trim/NativeAOT safety guidance (focused on IL2026/IL3050 annotation-chain analysis) motivated by the HybridWebViewHandler IL3050 incident.
Changes:
- Added Section 23 (“Trim / NativeAOT Safety”) to
review-rules.mdwith six concrete review checks to discourage suppression-based “fixes” and encourage correct structural patterns. - Updated
SKILL.mdto reflect the expanded rule set and added a new “Safety” category callout.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/skills/code-review/references/review-rules.md | Adds a new Section 23 with reviewer heuristics for Trim/NativeAOT (IL2026/IL3050) issues and recommended fix patterns. |
| .github/skills/code-review/SKILL.md | Updates the skill metadata/description and lists “Safety” as a review category referencing the new section. |
| |-------|-----------------| | ||
| | **`[UnconditionalSuppressMessage]` for IL2026/IL3050 is almost always wrong** | If a PR adds `[UnconditionalSuppressMessage("AOT", "IL3050:...")]` or `[UnconditionalSuppressMessage("Trimming", "IL2026:...")]`, it is very likely hiding a real problem rather than fixing it. The correct response is to restructure the code so the analyzer can prove safety — typically via the extract-method pattern below. Only accept suppression if the PR includes proof that the code path is genuinely unconditionally safe (e.g., the type is explicitly preserved). (Issue #34867 — simonrozsival) | | ||
| | **`[FeatureGuard]` does NOT suppress indirect annotation chains** | `[FeatureGuard(typeof(RequiresDynamicCodeAttribute))]` on a `RuntimeFeature` property suppresses IL3050 only for **direct** calls to `[RequiresDynamicCode]` methods inside the guarded block. It does **not** suppress warnings from indirect chains — e.g., generic type parameters with `[DynamicallyAccessedMembers]` that trace back to an annotated type. If a PR claims "the feature guard handles it" but the warning goes through a generic like `AddHandler<T, TRender>()`, the guard is NOT sufficient. (Issue #34867 — simonrozsival) | | ||
| | **Extract-method pattern for feature-guarded annotated types** | When registering a type annotated with `[RequiresDynamicCode]` or `[RequiresUnreferencedCode]` inside a `[FeatureGuard]`-protected block, and the registration goes through a generic API with `[DynamicallyAccessedMembers]`, extract the call into a separate method annotated with the same attributes. This converts the indirect chain into a direct call that the `[FeatureGuard]` can suppress. Reference: `AppHostBuilderExtensions.AddHybridWebViewHandler()` in `src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs`. (PR #34958 — simonrozsival) | |
There was a problem hiding this comment.
The reference to AppHostBuilderExtensions.AddHybridWebViewHandler() appears to be stale/incorrect: there is no AddHybridWebViewHandler method in src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs (current code registers HybridWebView inside AddControlsHandlers / SetupDefaults). Please update this reference to the actual helper/method name used in the repo (or link to a stable permalink to the relevant PR/commit) so reviewers can find the example quickly.
| | **Extract-method pattern for feature-guarded annotated types** | When registering a type annotated with `[RequiresDynamicCode]` or `[RequiresUnreferencedCode]` inside a `[FeatureGuard]`-protected block, and the registration goes through a generic API with `[DynamicallyAccessedMembers]`, extract the call into a separate method annotated with the same attributes. This converts the indirect chain into a direct call that the `[FeatureGuard]` can suppress. Reference: `AppHostBuilderExtensions.AddHybridWebViewHandler()` in `src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs`. (PR #34958 — simonrozsival) | | |
| | **Extract-method pattern for feature-guarded annotated types** | When registering a type annotated with `[RequiresDynamicCode]` or `[RequiresUnreferencedCode]` inside a `[FeatureGuard]`-protected block, and the registration goes through a generic API with `[DynamicallyAccessedMembers]`, extract the call into a separate method annotated with the same attributes. This converts the indirect chain into a direct call that the `[FeatureGuard]` can suppress. Reference: `HybridWebView` registration in `AddControlsHandlers` / `SetupDefaults` in `src/Controls/src/Core/Hosting/AppHostBuilderExtensions.cs`. (PR #34958 — simonrozsival) | |
| description: >- | ||
| Deep code review of PR changes for correctness, safety, and MAUI conventions. | ||
| Uses independence-first assessment (code before narrative) with 345 lines of | ||
| Uses independence-first assessment (code before narrative) with 364 lines of |
There was a problem hiding this comment.
The SKILL description says the rules file has "364 lines", but references/review-rules.md currently has 365 lines. Consider either updating the number or (more robust) removing the hard-coded line count so it doesn't drift each time rules are edited.
| Uses independence-first assessment (code before narrative) with 364 lines of | |
| Uses independence-first assessment (code before narrative) with |
|
/review -b feature/refactor-copilot-yml |
This comment has been minimized.
This comment has been minimized.
|
/review -b feature/enhanced-reviewer -p android |
|
|
kubaflo
left a comment
There was a problem hiding this comment.
🤖 Multi-model code review — Request changes
Three models reviewed this PR independently (Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro), then cross-pollinated.
Consensus: ❌ Request changes — valuable section, but two technical inaccuracies in a skill that shapes future reviews.
The core guidance is sound ✅
The headline technical claim — that [FeatureGuard(typeof(RequiresDynamicCodeAttribute))] suppresses IL3050 only for direct [RequiresDynamicCode] calls, not for indirect chains (generic params with [DynamicallyAccessedMembers] that trace back to an annotated type) — is correct (all three models independently verified it against how the trimmer/ILC propagates annotations). The "suppression is almost always the wrong fix" framing and the #pragma warning disable ILxxxx equivalence are also fair. The line-count bump (364) and markdown are fine.
Two accuracy fixes before merge
- Branch-mismatched code reference (line 344) — the extract-method example cites
AppHostBuilderExtensions.AddHybridWebViewHandler(), but that method exists only on net11.0 (where the fix PR #34958 merged), not on this PR'smainbase — here the registration is still inline (AddHandler<HybridWebView, HybridWebViewHandler>()inside theRuntimeFeature.IsHybridWebViewSupportedguard, ~lines 130-133). A rule added tomainthat points at code absent frommainwill confuse reviewers — update it to matchmain(or note it's net11.0-only / pending #34958). - Incorrect IL claim (line 345) — "accessing a
conston the type embeds the type reference in the caller's IL metadata" is false: aconst stringis folded to anldstrliteral at compile time with noTypeRef/MemberRefto the declaring type (Opus also noted the originating commit 2 was a cosmetic "address review" edit, not a trimmer fix). The "use a local const instead" advice is harmless but its stated reason is a misconception (it would be true for astatic readonlyfield, not aconst).
Since this is the code-review skill itself, getting the rules technically exact matters — otherwise reviewers will cite phantom code or flag non-issues.
Independent verdicts: GPT-5.5 — NEEDS_CHANGES (high) · Gemini 3.1 Pro — NEEDS_CHANGES · Opus 4.8 — NEEDS_DISCUSSION (high). Consensus: keep the section (the FeatureGuard insight is genuinely useful and was empirically verified), fix the two inaccuracies.
118f3a1 to
767cbb3
Compare
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ## Summary - Upgrade the Skill Validation workflow from `@microsoft/vally-cli@0.6.0` to `0.10.0`. - Preserve MAUI's explicit `*.vally.yaml` enumeration because Vally directory discovery recognizes only `eval.yaml` and `eval.yml` by default. - Update all 0.6-specific scoring guidance: these specs intentionally omit `scoring.weights`, retaining Vally 0.10's equal-weight aggregation and existing thresholds. ## Validation - Ran strict lint with `@microsoft/vally-cli@0.10.0` for all 10 MAUI eval specs. - Parsed `.github/workflows/skill-validation.yml` as YAML and ran `git diff --check`. ## Scope This PR is limited to the Vally 0.10.0 upgrade. It intentionally excludes the Trim/AOT fixture-workflow changes from #34962. Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com>
Add Section 23 to review-rules.md covering IL2026/IL3050 annotation chain analysis, the extract-method pattern for feature-guarded types, and why [UnconditionalSuppressMessage] is almost always wrong for these warnings. Sourced from the HybridWebViewHandler incident (Issue #34867, PR #34958) where AI agents consistently recommended suppression instead of the correct structural fix when they didn't read the source code first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
Replace workflow-generated repositories with inert Vally file fixtures and include the complete HybridWebView annotation chain for review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
60edf92 to
967d73c
Compare
Raise only the multi-surface prior-review capability case from five to ten minutes after a successful semantic run was terminated at the five-minute cap. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bf0a7e5c-7cad-457d-9f55-866aec36fe96
| # This live case traverses several GitHub review surfaces and routinely | ||
| # approaches the suite's five-minute default even when it succeeds. | ||
| max_duration: 10m | ||
| expect_skills: | ||
| - code-review |
kubaflo
left a comment
There was a problem hiding this comment.
🔍 AI-generated review (multi-model pipeline: Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), on behalf of @kubaflo — round 8 at
08cec5aa.
Multi-model review — PR #34962 (Trim/NativeAOT capability suite for code-review)
Three models reviewed the new hermetic Trim/AOT eval independently, then cross-pollinated. I (orchestrator) then verified every disputed claim myself against the actual Vally engine source and the fixture files.
What this adds
A hermetic code-review-trim-aot capability suite (eval.trim-aot.vally.yaml, 3 scenarios) + fixtures/trim-aot/** inert source snapshots + a maui-expert-reviewer.md touch-up:
- case-a — a justified, narrowly-scoped
IL2026/IL3050suppression behind theRuntimeFeature.IsHybridWebViewSupported[FeatureGuard](expect LGTM/NEEDS_DISCUSSION). - case-b — annotated-helper structural isolation of the dynamic path (expect LGTM/NEEDS_DISCUSSION).
- case-c — synthetic counterfactual: the guard becomes
if (true), so the pragma now hides a reachable warning (expect NEEDS_CHANGES).
The counterfactual + annotation-chain rubrics are genuinely well-designed. Nice suite.
⚠️ One real, non-blocking issue — the tool-calls "review-input access" grader is inert
eval.trim-aot.vally.yaml:72-82 (reused via the *review-input-only alias at :157 and :231)
The five required entries use a pattern: key to demand the agent actually read each file. But Vally's tool-calls matcher (ToolMatch) has no pattern key — it supports only name / command / path / args / result / min_count / final / before_step / at_step. I verified this directly against @microsoft/vally@0.8.0 (dist/graders/static/tool-call-grader.d.ts and the normalizeEntry impl, which reads only ToolArgKeys = ["command","path"] plus name/args/result).
Consequence: the unknown pattern: field is silently dropped, so all five required entries collapse to the identical name-only matcher { name: /^(bash|view|grep|rg)$/ }. Because required doesn't need distinct calls, a single bash/view/grep/rg call — the prompt-mandated read of change.diff — satisfies all five. The grader therefore never verifies the agent inspected the four source snapshots, which is precisely the annotation-chain tracing this suite exists to reward. A shortcutting agent that reads only the diff and guesses the verdict still passes this grader. It's also not caught by vally lint --strict (the toolCalls schema declares required/disallowed as bare arrays with no items, so entry shapes aren't validated).
Fix: express file access with supported keys — e.g. command: 'change\.diff' for the bash read, path: 'AppHostBuilderExtensions\.cs' for view/grep, or the generic args: map. (The disallowed: block is fine — it uses command: + name-only matchers.) CI pins @microsoft/vally-cli@0.10.0 (not on public npm), so worth confirming there — but the documented ToolMatch API presents args as the generalization of command/path, so a new top-level pattern matcher is unlikely.
✅ Investigated and rejected — "the change.diff files don't apply cleanly" (false positive)
Two models flagged that the change.diff hunk headers (@@ -129 @@, @@ -120 @@, @@ -126 @@) don't line up with the truncated ~24-line sibling snippets, or that leading context lines (Switch/Stepper) are missing — concluding the patches "won't apply cleanly." I checked this and it does not apply: the suite header states Vally materializes the patch + snapshots as inert fixture files, and each prompt asks the agent to "inspect change.diff, then read the supporting source snapshots" — nothing runs git apply/patch. I confirmed all three sibling AppHostBuilderExtensions.cs snapshots are the coherent post-change state of their diff (case-a L16-18, case-b L11+14-26, case-c L9-18 each equal the diff's + side). Real full-file hunk headers are correct and realistic here; rewriting them to match the snippet would make the fixtures less faithful. No change needed.
💡 Minor (optional)
.gitattributescould add*.diff text eol=lffor cross-platform determinism — cosmetic here since the diffs are read, not applied.- The
output-matchesverdict graders are unanchored regexes ((LGTM|NEEDS_DISCUSSION)); anchoring to the final verdict line would avoid matching the word in prose. (Pre-existing pattern across the suite — non-blocking.)
CI
license/cla ✅ pass · maui-pr skipping (expected — skills/eval-only PR, no product build; the eval runs in the separate skill-eval workflow, not maui-pr).
Verdict: NEEDS_CHANGES — single non-blocking test-quality fix
Swap the tool-calls pattern: keys for command:/path:/args: so the access grader actually enforces reading the four snapshots, and this is good to go. The two "won't-apply" findings were verified false positives.
Consensus: Opus 4.8 → NEEDS_CHANGES (inert grader — the real find); GPT-5.5 → NEEDS_CHANGES (apply concern, false positive); Gemini 3.1 Pro → NEEDS_CHANGES (apply concern, false positive). Orchestrator independently confirmed the grader defect against vally@0.8.0 source and rejected the apply concern as a non-issue.
| config: &review-input-only | ||
| required: | ||
| - name: '^(bash|view|grep|rg)$' | ||
| pattern: 'change\.diff' |
There was a problem hiding this comment.
🔍 AI-generated inline note (multi-model orchestrator, on behalf of @kubaflo)
This "review-input access" grader is inert. Vally's tool-calls ToolMatch has no pattern key — it supports only name/command/path/args/result/min_count/final (verified against @microsoft/vally@0.8.0 tool-call-grader.d.ts + normalizeEntry, which reads only ToolArgKeys=["command","path"] plus name/args/result). So pattern: here is silently dropped and all five required entries collapse to the same { name: /^(bash|view|grep|rg)$/ } matcher — one change.diff read satisfies all five, and the grader never enforces that the agent read the four source snapshots. Also reused via *review-input-only at lines 157 and 231.
Fix — use supported keys, e.g.:
required:
- name: '^(bash|view|grep|rg)$'
command: 'change\.diff'
- name: '^(view|grep|rg)$'
path: 'AppHostBuilderExtensions\.cs'
# …or the generic args: map(disallowed: below is fine — it uses command:/name-only.) CI pins vally-cli 0.10.0 (not on public npm) — worth confirming there, though args is documented as the generalization of command/path, so a new pattern matcher is unlikely.
kubaflo
left a comment
There was a problem hiding this comment.
🔍 AI-generated review (GitHub Copilot CLI, on behalf of @kubaflo) — multi-model adversarial pass (Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), independent → cross-pollinated → synthesized. Round 9 at 08cec5aa.
✅ LGTM — Trim/NativeAOT safety rules land clean-green
The three models independently converged on approve. This round adds a dedicated Trim/AOT review capability and — with the follow-up commits — the whole code-review skill validation is green at head.
What this delivers
maui-expert-reviewer.md— 4 newIL2026/IL3050annotation-chain CHECK rules: a#pragma/suppression is only acceptable under a real runtime guard; a defeated guard (e.g.if (true)) or a structural fix that silently drops the annotation must be flagged as reachable.eval.trim-aot.vally.yaml(new suite, +275 lines) — a 3-fixture adversarial triad that is exactly the right shape to prevent both failure modes:- case-a —
#pragmaunder a genuine guard → accept ✅ - case-b —
[RequiresDynamicCode]structural fix → accept the structural change ✅ - case-c — guard defeated to
if (true)→ reject as reachable ✅ - Judge rubrics guard both directions (no over-reject of legitimate suppressions, no rubber-stamp over-accept of defeated guards).
- case-a —
CI — authoritative verdict at 08cec5aa
Read from the skill-validation results comment + aggregate check (not a single matrix leg):
-
Overall: Passed · LLM: Passed · aggregate "Skill Validation": success
-
4/4 eval suites met threshold:
Suite Score Threshold code-review-trim-aot (this PR) 1.00 0.90 ✅ code-review-inline-findings-write 1.00 0.70 ✅ code-review-capabilities 0.91 0.60 ✅ code-review-regressions 0.89 0.60 ✅ -
The follow-up commit "Allow live review reconciliation to finish" fixed the previously-flaky
prior-review-reconciliation-surfaces-prior-findingsscenario — now 3/3 (100%).
Residual (non-blocking)
- One unrelated pre-existing scenario,
verdict-consistency-errors-block-lgtm, shows a 33% minority flake (output-matches2/3,pass^k~29.6%). It is not touched by this PR (Trim/AOT rules + reconciliation fix), and its suite still passes with a wide margin (0.91 vs 0.60), so the aggregate stays green. Worth a future stabilization pass but not a blocker here.
Verdict: LGTM · confidence high
The Trim/AOT capability is a clean, well-designed addition with a perfect adversarial-triad eval, and the whole skill-validation suite is green at head. Nice work.
ℹ️ Independent multi-model review synthesized by GitHub Copilot CLI for @kubaflo. Not a substitute for human maintainer judgment.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Summary
Add Trim/NativeAOT annotation-chain guidance to the active MAUI expert reviewer and cover it with hermetic Vally scenarios.
Motivation
The HybridWebView NativeAOT investigation showed that IL2026/IL3050 warnings require annotation-chain and target-toolchain analysis. A blanket rule that every suppression is wrong is inaccurate: #34868 uses a documented, narrowly scoped Android NativeAOT ILC workaround, while #34958 demonstrates the structural annotated-helper alternative on
net11.0.What changed
maui-expert-reviewer.md: expands the Trimming/AOT dimension to require tracing the annotated member, generic/DAM hops, feature guard, reachability, and target analyzer. It distinguishes broad suppression from a documented, scoped toolchain workaround and covers structural-helper/source-generation alternatives.environment.filesto materialize frozen candidate diffs and the exact supporting source snapshots needed to trace the HybridWebView handler annotations, generic registration DAM hop, and runtime feature guard. No Git history, remote repository, or custom workflow fixture setup is exposed to the reviewer.if (true)and must be rejected.Evaluation scope
The separate
eval.trim-aot.vally.yamlsuite tests technical review reasoning on inert checked-in inputs. Tool-call grading requires inspection of the candidate patch and all annotation-chain sources while rejecting GitHub/network access. The five-run suite retains a0.9threshold so the required verdict cannot be masked by the other graders.