Skip to content

feat(action): opt-in resolution of the bot's own outdated review threads (#567) - #944

Merged
lizhengfeng101 merged 2 commits into
alibaba:mainfrom
chethanuk:feat/567-resolve-outdated-threads
Sep 17, 2026
Merged

lizhengfeng101 merged 2 commits into
alibaba:mainfrom
chethanuk:feat/567-resolve-outdated-threads

Conversation

@chethanuk

@chethanuk chethanuk commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

Closes #567. When OCR comments inline and the author pushes a fix, the conversations have to be resolved by hand. This adds a resolve_outdated input, default off, that resolves review threads GitHub has already marked outdated.

Nothing here asks a model. @lizhengfeng101's objection on the issue — that inferring "was this addressed?" from a later diff is hallucination-prone, and that tolerance for false positives at the last gate before merge is near zero — rules that out. The signal is GitHub's own server-computed isOutdated, which it recomputes against the current head and which is force-push and rebase aware.

@stay-foolish-forever asked on the issue how conservative this should be. isOutdated is necessary but not sufficient, so four vetoes sit on top of it. Each can only refuse to resolve a thread, never cause one to be resolved:

thread marked isOutdated by GitHub
  │
  ├─ carries OCR's comment marker? ─────────── no ──► skip: not_ours
  ├─ every comment from the root's author? ─── no ──► skip: human_reply
  ├─ original line resolvable? ─────────────── no ──► skip: unverified
  ├─ any current finding touches those lines? ─ yes ─► skip: overlap
  └─ this run parsed ≥1 finding? ───────────── no ──► skip: no_findings
      │
      └──► resolveReviewThread   (never minimize, never delete)

Three modes: 'false' (default, no GraphQL calls at all), 'report' (runs the whole gate, logs each thread's verdict, mutates nothing), 'true'.

Two places this departs from the obvious implementation

Identity is read off the node the marker proved, not off users.getAuthenticated(). That endpoint is user-token-only and 403s under every Actions token, including the default GITHUB_TOKEN — so an implementation that trusts it gets botLogin: null on every real run and falls back to a hard-coded github-actions[bot] match. Under any custom GitHub App token that classifies OCR's own comments as human replies: the feature resolves nothing and logs skipped=human_reply:N, which an operator reads as "humans replied to all your threads". The gate now takes the root comment's author as the identity once the marker has proved the thread is ours, and requires every reply to match it. That also closes the PAT case, where the login is a human's and their own replies would otherwise have counted as the bot's.

The overlap veto uses plain span intersection, not sameCommentSpan. Reusing the incremental-dedupe predicate looked right and is backwards here: its first rule is cur.multiline !== other.multiline → false, which is correct when a finer single-line note should not be suppressed by an old block comment, and wrong when the question is "does anything current still touch these lines?". It let three real overlaps through — a single-line thread against a covering multi-line finding, a multi-line thread against a single-line finding inside it, and a thread strictly contained by a current finding. All three now veto. incremental_overlap_threshold is deliberately not consulted: an IoU threshold is a similarity test, and this gate needs "any shared line at all".

Limitations

  • contents: write is required for resolveReviewThread, measured rather than assumed. On a pull_request_target workflow that is a real trade, and the README says so rather than burying it.
  • 'report' mode does not exercise that permission, so a healthy preview can still be followed by FORBIDDEN on flipping to 'true'. Documented at the input.
  • 50 threads per run, ~100s at the existing OCR_SUCCESS_DELAY pacing. No new env knob — the one this originally added duplicated OCR_SUCCESS_DELAY's job and is gone.
  • Resolving an outdated thread is not the same claim as "this was fixed". A fix applied elsewhere leaves the thread open; a refactor that moves untouched buggy code closes it. isOutdated means the flagged lines left the diff, and the vetoes exist so that stays the only claim being made.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no functional changes)
  • Documentation update
  • CI / Build / Tooling

How Has This Been Tested?

  • make test passes locally
  • Manual testing (described below)

node scripts/github-actions/post-review-comments.test.js — all green, including the pre-existing suites. node scripts/github-actions/check-translation-sync.test.js green. action.yml parses.

The core is a 22-case table over the resolve predicate covering each veto and each ordering between them. On top of that, four cases pin the two behaviors above, each written to fail against the obvious wrong implementation:

  • identity derived from the root author, and read off the node the marker proved — both fail if getAuthenticated()'s result is trusted;
  • the veto flipped to plain intersection — fails against sameCommentSpan, and covers single-vs-multi and containment, which the old expectations had codified the wrong way round;
  • action.yml does not promise incremental_overlap_threshold to resolve_outdated, asserted against the file so the docs cannot drift back.

The identity mock now rejects with 403, which is what production actually returns. Every prior test passed a login that no Actions token can obtain, which is precisely what hid the bug.

With the feature off, all five pre-existing comments_* outputs are asserted byte-identical and no GraphQL call is made.

Checklist

  • My code follows the project's coding style (go fmt, go vet)
  • I have performed a self-review of my code
  • I have added tests that prove my fix is effective or my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly (if applicable)
  • I have signed the CLA

Related Issues

Closes #567

AI/LLM Disclosure

Written with AI assistance using PI Agent (Muse Spark 1.3).

@github-actions

github-actions Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 3 issue(s) in this PR.

  • ✅ Successfully posted inline: 3 comment(s)

Comment on lines +132 to +133
const resolveMode =
resolveOutdated === "true" ? "resolve" : resolveOutdated === "report" ? "report" : "off";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style · low
This is a nested ternary expression (resolveOutdated === "true" ? "resolve" : resolveOutdated === "report" ? "report" : "off"), which is disallowed per the coding standards. Consider using a simple lookup object or an if/else chain for clarity and future extensibility (e.g., if more modes are added later).

Suggestion:

Suggested change
const resolveMode =
resolveOutdated === "true" ? "resolve" : resolveOutdated === "report" ? "report" : "off";
const RESOLVE_MODES = { true: "resolve", report: "report" };
const resolveMode = RESOLVE_MODES[resolveOutdated] || "off";

Comment on lines +855 to +856
out("comments_resolved", String(stats.resolved || 0));
out("comments_resolved_preview", String(stats.resolvedPreview || 0));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
The || 0 guards here are redundant: stats.resolved and stats.resolvedPreview are always initialized to 0 in the stats object (lines 120-121) before any code path reaches setStatsOutputs. The || 0 makes these look like they could be undefined, which they cannot be. This is harmless but slightly misleading for future maintainers.

Suggestion:

Suggested change
out("comments_resolved", String(stats.resolved || 0));
out("comments_resolved_preview", String(stats.resolvedPreview || 0));
out("comments_resolved", String(stats.resolved));
out("comments_resolved_preview", String(stats.resolvedPreview));

Comment on lines +1246 to +1249
function threadIsOurs(thread) {
const root = rootComment(thread);
return new RegExp(OCR_COMMENT_ID_SOURCE).test((root && root.body) || "");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance · low
A new RegExp object is compiled from OCR_COMMENT_ID_SOURCE on every invocation of threadIsOurs. Since this function is called once per thread in the listing (potentially hundreds of times), and the regex is stateless (no /g flag), it can be compiled once at module scope and reused. The code already correctly notes in getPostedCommentIds that /g carries lastIndex and must not be shared — but a non-/g regex is safe to share across calls.

Suggestion:

Suggested change
function threadIsOurs(thread) {
const root = rootComment(thread);
return new RegExp(OCR_COMMENT_ID_SOURCE).test((root && root.body) || "");
}
const OCR_ID_RE = new RegExp(OCR_COMMENT_ID_SOURCE);
function threadIsOurs(thread) {
const root = rootComment(thread);
return OCR_ID_RE.test((root && root.body) || "");
}

@chethanuk
chethanuk force-pushed the feat/567-resolve-outdated-threads branch from 38a395e to e2de6f4 Compare August 21, 2026 06:03
@chethanuk

Copy link
Copy Markdown
Contributor Author

@lizhengfeng101 Rebased onto main and conflicts resolved. Please review and merge this. Thanks :)

@chethanuk

Copy link
Copy Markdown
Contributor Author

@lizhengfeng101 Rebased onto main and conflicts resolved again. Please review this. Thanks :)

@lizhengfeng101

Copy link
Copy Markdown
Contributor

I'd like to settle the permission question before the rest, because it's the part that worries me most.

Asking adopters to add contents: write to a pull_request_target workflow that checks out untrusted fork code is a real widening of the blast radius — our sample workflow runs contents: read on purpose. And resolving a review thread is pull-request conversation state, so I'd expect pull-requests: write to be enough; contents covers files, branches and tags.

Could you share the measurement behind contents: write? If pull-requests: write turns out to be sufficient, both the docs and the runtime warning need to change. If it genuinely is required, that's counter-intuitive enough to be worth a code comment with the evidence. I couldn't verify it from the docs myself, which is why I'm asking rather than asserting.

@lizhengfeng101

Copy link
Copy Markdown
Contributor

Separately from the permission thread — I read through the rest closely, and the shape of the design holds up: no model in the decision, isOutdated as the necessary condition, every additional check able only to refuse. A few things I'd want to see addressed before merge.

A single no-line finding switches the whole feature off. The unlocatable guard fires on exactly the set of findings that already go to the summary via commentsWithoutLine — path present, line missing. Those are normal output, not an anomaly, so one of them means the run resolves nothing and doesn't even list threads. Fail-closed is the right instinct, but the granularity could be finer: when the path is known and only the line is missing, veto that path; keep the run-wide disable for findings with no path at all. As written, I'd expect this to quietly no-op on a good share of real runs.

spansIntersect compares paths as exact strings. Your own comment notes that result.comments arrives from the model with no normalization. overlapsHistory does the same, but there a mismatch costs a duplicate comment, whereas here it costs a resolved thread sitting on top of a live finding. A light normalization, or at least a note that the gap is known and accepted, would help.

getAuthenticatedLogin runs twice when incremental and resolve_outdated are both on (:358 and :525) — two 403s, and two [incremental] could not resolve authenticated user lines, the second from a path that has nothing to do with incremental mode. Worth hoisting, and the prefix is now misleading.

resolve_outdated is validated in JS, while effort is validated in the Validate inputs step. A typo currently costs a full LLM run before anyone hears about it. Falling back to off is the right direction, but failing fast would match how the other enum inputs behave.

And per AGENTS.md #1, the description still needs the AI/LLM disclosure — which tools and models. Everything else in the checklist is filled in.

Smaller things I wouldn't hold this up over: comments_resolved_preview is the uncapped candidate count while action.yml describes it as what would have been resolved; unverified covers two distinct causes and report mode is the only diagnostic surface there is; the output list further down the README doesn't mention the two new outputs; and the U1/U2 test labels point at an audit process that won't mean much to a reader six months from now.

For what it's worth, I ran npm run test:github-actions against your head — all four suites green.

@lizhengfeng101

Copy link
Copy Markdown
Contributor

@chethanuk

…ads (alibaba#567)

Adds a resolve_outdated input, default off, that resolves review threads
GitHub has already marked outdated. No model is involved in the decision:
the signal is GitHub's server-computed isOutdated, and five vetoes sit on
top of it, each of which can only refuse to resolve a thread.

Three modes: 'false' (no GraphQL calls at all), 'report' (log the verdict
for every thread, mutate nothing), 'true'.
@chethanuk
chethanuk force-pushed the feat/567-resolve-outdated-threads branch from 4ff6066 to f2b4342 Compare September 17, 2026 10:15
@chethanuk

Copy link
Copy Markdown
Contributor Author

Rebased onto main, merge commit gone.

Could you share the measurement behind contents: write?

https://github.com/chethanuk/open-code-review/actions/runs/35202780271 — three jobs, identical but for permissions:, each resolving its own thread, no try/catch:

contents pull-requests Result
read write FORBIDDEN
write read resolved
write write resolved

Necessary and sufficient, and pull-requests: write is neither. The mutation isn't gated on the PR scope at all: you can resolve a conversation if you opened the PR or have write access to the repository, and for an installation token that comes from Contents. Run URL and reasoning are now in a comment above the mutation.

Asking adopters to add contents: write to a pull_request_target workflow that checks out untrusted fork code is a real widening of the blast radius

Agreed, and that's what the docs got wrong. The README now shows a split-job layout: the review job stays contents: read, and a second job with no actions/checkout holds the elevated scope.

when the path is known and only the line is missing, veto that path; keep the run-wide disable for findings with no path at all

Done, exactly that.

spansIntersect compares paths as exact strings.

Normalized on both sides. I left overlapsHistory alone on purpose — same gap, but it's incremental-mode behavior and this PR isn't about that. Separate PR if you'd rather.

getAuthenticatedLogin runs twice [...] Worth hoisting, and the prefix is now misleading.

One memoized call, and the prefix is no longer [incremental]. I kept the second call site: botLogin still gates the PAT case, where the root comment's author is a User.

failing fast would match how the other enum inputs behave

Moved into Validate inputs, same shape as stream_progress. The JS fallback stays for direct callers.

- Veto by path instead of disabling the whole run: a finding with a path but
  no line is ordinary output and now costs only its own path, while a finding
  with no path at all still disables the run.
- Normalize paths before the overlap veto, so ./src/a.js and src/a.js no
  longer compare unequal and resolve a thread over a live finding.
- Resolve the authenticated login once per run rather than once per caller.
- Validate resolve_outdated in the Validate inputs step, so a typo fails
  before the LLM runs rather than silently falling back to off.
- Correct the permission documentation from measurement, and show the
  contents: write scope in a job that never checks out the pull request.
- Split the overloaded unverified reason, fix the documented preview count,
  list the missing outputs, and replace the nested ternary with a Map.
@chethanuk
chethanuk force-pushed the feat/567-resolve-outdated-threads branch from f2b4342 to c29e821 Compare September 17, 2026 11:06

@lizhengfeng101 lizhengfeng101 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work!

@lizhengfeng101
lizhengfeng101 merged commit cb6ea26 into alibaba:main Sep 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI Actions to automatically resolve past comments

2 participants