Skip to content

Fix visual review capture contract and incomplete-report processing - #76585

Merged
pelikhan merged 5 commits into
mainfrom
pelikhan/visual-review-reporting-failure
Nov 10, 2026
Merged

pelikhan merged 5 commits into
mainfrom
pelikhan/visual-review-reporting-failure

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Summary

Fixes the two distinct boundaries observed in Visual Regression Checker run 42849185332, relating to #76555:

  • Serve setup-provisioned docs build artifacts inside the sandbox on loopback instead of starting a runner-host server. Remove the host-domain/curl exception, use the pinned Playwright CLI's actual commands, and distinguish unavailable baseline comparison from failed screenshot/accessibility capture.
  • Exempt only the exact built-in local-only report_incomplete handler from unnecessary PR lifecycle reads, using private function-identity registration. Custom replacements and every mutating handler retain fresh fail-closed checks in initial and deferred processing.
  • Preserve sanitized HTTP status and GitHub request identity when API errors have an empty message. No whole-handler/script retry or replay of the already-published comment.

Evidence and validation

  • Failed execution was PR merge 066a23d0ff56421f5ed7d43475145a7d63845e4c, not evidence of a current-main regression. The agent stopped before navigation; later PR GET 500 occurred after successful comment publication.
  • Six focused JavaScript suites: 912 passed, 3 skipped. Covers local diagnostics, once-only comment publication, mutation fail-closed behavior, same-named custom overrides, deferred closure checks, empty HTTP errors, conclusion incompletion handling, and visual workflow contract.
  • make fmt, make build, make recompile, static typecheck, and final make agent-report-progress passed. Final workflow drift check compiled 365 workflows; existing repository-wide warnings remain.
  • Diagnostic dry-run compilation passed source/lint/model checks and shellcheck. Diagnostic lock preserved privately; reviewed production lock restored. Optional scanners not run.
  • Single read-only security reviewer: PASS, all ten required questions YES, no confirmed vulnerabilities remaining. Reviewed source and production lock hashes preserved through publication.

Overlap check and scope

Checked open PR inventory and candidate file patches before implementation and again before publication. #76537 (now merged) changes tool lists only; #76425 touches a different manager claim-settlement path; #76538 concerns CLI-proxy capture, and #76580 provisioning does not touch these paths. No equivalent fix found; generated-lock/shared-file merge conflicts are possible, but their behavior is not duplicated here.

No workflow dispatch/rerun or production-write replay performed. Actual hosted preview startup, permission matching (owned by separate work), browser rendering/proxy/reachability, and screenshot output remain unverified; this PR does not claim live recovery.

Authored Markdown security overview

Source Change Security impact
.github/workflows/visual-regression-checker.md frontmatter/setup Remove runner-host server and host exception; retain dependency provisioning/build Narrower network/command access; credentials, read permissions and one-comment cap unchanged
Workflow prompt Sandbox-loopback preview, supported CLI examples, honest baseline handling and process cleanup Preserves browser policy without runtime package installs, environment inspection or proxy changes

Keep browser capture on sandbox loopback with provisioned build artifacts and supported CLI commands. Preserve unavailable baseline status without blocking capture. Limit the lifecycle-read exemption to the built-in diagnostic handler identity, and retain HTTP/request context for empty API errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI dynamic review requested due to automatic review settings November 10, 2026 19:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

SDK permissions block the readiness probe, contract tests still require the removed setup step, and substring-based error classification can incorrectly turn publication failures into successful skips.

3 open issues
What changed in this PR

Updates gh-aw’s visual review workflow and safe-output processing to address the capture and incomplete-report failures described in #76555.

Changes:

  • Moves preview startup into the sandbox, corrects Playwright CLI instructions, and distinguishes missing baselines from capture failures.
  • Exempts only the built-in report_incomplete handler from PR-state reads while retaining checks for custom and mutating handlers.
  • Adds sanitized fallback API diagnostics, regression coverage, and a regenerated workflow lock.
File Description
actions/​setup/​js/​visual_regression_workflow.test.cjs Adds workflow contract checks for preview, capture, baseline handling, and isolation.
actions/​setup/​js/​safe_output_handler_manager.test.cjs Covers diagnostic exemptions, once-only publication, and custom/deferred handler checks.
actions/​setup/​js/​safe_output_handler_manager.cjs Registers the built-in diagnostic exemption by function identity.
actions/​setup/​js/​error_helpers.test.cjs Tests empty-message fallbacks and sanitized request IDs.
actions/​setup/​js/​error_helpers.cjs Adds HTTP status and request identity to empty-error diagnostics.
.github/​workflows/​visual-regression-checker.md Replaces runner-host serving with sandbox-local preview and revised capture instructions.
.github/​workflows/​visual-regression-checker.lock.yml Regenerates execution configuration without the host exception or server setup steps.

Review effort: Max (auto)

Comment thread actions/setup/js/error_helpers.cjs Resolved
Update build-only checkout contracts and grant the exact bounded readiness command. Classify API errors using structured HTTP status so request IDs cannot mask publication failures or produce incorrect authentication guidance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Fix failing check Native JavaScript and Actions parser tests (FAILURE): Check run.

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "87ee21a1f620599c1f9fa2f9c20eebda53a2b0b8",
  "kind": "requested",
  "schema_version": "1",
  "state": "9d1852961bf3646db92495ddba876cd3ecdc2c47c1f0ac4266b1c073ced8e0f5",
  "work": [
    "b855fc6fcb55b78d8a97b8924885cae846819f33bbc1dec6e0b470fbcf24991c"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits November 10, 2026 20:22
…w-reporting-failure

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Merged the latest main in 3a03f99. Fixed the failing parser check in 657da81 by updating the JavaScript summary catalog digest for error_helpers.cjs; the full pkg/workflow/taint suite and final repository validation passed.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "657da81b8d991730494f2a5f3e119097620b2e76",
  "kind": "requested",
  "schema_version": "1",
  "state": "b339c51b9fb8beae1023db56a6c3d5b0b478aa4dcad3d3d95b77d94369f65ca9",
  "work": [],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

@pelikhan

Copy link
Copy Markdown
Collaborator Author

@copilot resolve the merge conflicts on this branch.

…w-reporting-failure

# Conflicts:
#	.github/workflows/visual-regression-checker.lock.yml

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Merged the latest main in 7d885b4. The generated visual-regression lock conflict was resolved by recompiling from the merged Markdown; build, recompile, formatting, lint, and impacted tests passed.

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts on this branch.

The merge conflicts were resolved in 7d885b4. The merge commit has both the previous branch head and main as parents, and there are no unresolved files or conflict markers.

@pelikhan
pelikhan merged commit 77de61c into main Nov 10, 2026
66 checks passed
@pelikhan
pelikhan deleted the pelikhan/visual-review-reporting-failure branch November 10, 2026 23:22
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants