Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/aw/debug-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,8 @@ rather than skipping them; review callees separately without the override.
push jobs, memory/cache persistence, reusable safe-output calls, work-queue
operations, reactions, status/failure comments and issues, label removal, and
issue locking. Diagnostics, summaries, and run artifacts remain available;
`OTEL_*` and `GH_AW_OTLP_*` environment variables and automatic telemetry
`OTEL_*`, `GH_AW_OTLP_*`, `GITHUB_AW_OTEL_*`, `TRACEPARENT`, `TRACESTATE`,
`BAGGAGE`, and `CLAUDE_CODE_ENABLE_TELEMETRY` environment variables and automatic telemetry
configuration/authentication are omitted from dry-run locks. Custom scripts
that configure their own exporters remain outside this control.
Review that emitted lock, not the previously published normal lock. If a live
Expand Down
88 changes: 56 additions & 32 deletions .github/aw/debug-security-review.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Gate local debug/dry-run changes with 2-3 independent cheap security judges.
description: Gate local debug/dry-run changes with one read-only security reviewer and an annotated Markdown-source overview.
disable-model-invocation: true
---

Expand Down Expand Up @@ -29,12 +29,12 @@ security checks. A passing dry-run still grants no live-dispatch permission.
## Agent Ownership

The agent performs the review: capture intent, inspect source and reachable
runtime dependencies, prepare the evidence packet, run independent judges,
runtime dependencies, prepare the evidence packet, engage one security reviewer,
resolve findings and missing coverage, execute permitted compilation/scanners,
and record the result. Do not hand these steps to the user as a checklist.
An incomplete judge response is missing review evidence, not a confirmed
An incomplete reviewer response is missing review evidence, not a confirmed
vulnerability or a reason to ask the user to inspect code. Gather the missing
evidence within the judge limits below; never turn UNKNOWN into PASS.
evidence with the same reviewer; never turn UNKNOWN into PASS.

The agent also prepares exact revisions, hashes, credential-source/destination
metadata, permissions, protections and run bounds for any required human
Expand Down Expand Up @@ -91,7 +91,7 @@ The agent may run the installed, trusted `gh aw compile WORKFLOW --dry-run`
to obtain emitted locks and compiler/scanner evidence before the final
source/lock review. This permission covers compilation, not execution of
workflow scripts, changed compiler code, build hooks, harnesses or MCP servers.
Review those separately before running them. The judges remain read-only.
Review those separately before running them. The reviewer remains read-only.

Emission is required: do not combine `--dry-run` with `--no-emit`. Compilation
may run in the current checkout; a disposable checkout is optional, not a
Expand All @@ -113,8 +113,8 @@ the diagnostic lock's review verdict to a different live lock.
request: goal, permitted scope/effects, preserved behavior and acceptance
criteria. For an existing session, capture before further edits/execution.
- Cite the originating user message and subsequent explicit user clarifications.
Store the concise intent record in session artifacts; give every judge the
same record. Do not copy unrelated or sensitive conversation content.
Store the concise intent record in session artifacts; give the reviewer that
record. Do not copy unrelated or sensitive conversation content.
- Clarification optional: if goal/scope/authorization is unclear and the user is
available, ask one focused question. Do not wait indefinitely for an absent
user. Record unresolved intent as UNKNOWN; continue static diagnosis and
Expand All @@ -139,24 +139,22 @@ the diagnostic lock's review verdict to a different live lock.
- No execution, installs, server startup or destination contact. Missing/truncated
relevant code = UNKNOWN coverage.

## Judges

- Select smallest/cheapest available code-classification models through supported
runtime controls. No guessed IDs or silent expensive defaults. Record actual
models; prefer distinct cheap models.
- Launch two independent read-only subagents in parallel: same packet/questions,
one bounded pass each, small response budget, runtime-supported timeout.
No nested agents, execution, external writes or shared answers.
- Disagreement/abstention: third independent cheap judge reviews disputed questions
against the same packet, blind to earlier votes. Maximum three judges/revision,
not per question. No favorable-vote retries.
- Cheap selection/read-only agents unavailable: gate UNAVAILABLE, never PASS.
## Single Reviewer

- Require exactly one read-only security-review agent for the current revision.
Give it the intent, evidence packet and questions below for one bounded pass.
No parallel judges, voting, tie-breakers, nested agents, execution or external writes.
- Use the configured runtime model defaults; do not guess or require model IDs.
Record the actual model if exposed, otherwise report it as unavailable.
- Resolve incomplete coverage or questions through bounded follow-ups with the
same reviewer. Do not launch replacement reviewers to obtain a favorable verdict.
- If a read-only reviewer is unavailable, the gate is UNAVAILABLE, never PASS.
Continue static diagnosis/editing; no unreviewed execution.

## Binary Questions

[BinEval-style](evals.md): one falsifiable claim/question; YES = property holds.
Judge code evidence, not reported success. Local prompts, not workflow `evals:`
Review code evidence, not reported success. Local prompts, not workflow `evals:`
or hosted runs. Scope every question to reachable changed behavior.

| ID | Question (YES = safe property) |
Expand All @@ -176,7 +174,7 @@ Seek exfiltration (encoded/covert included), malware, download-and-execute,
obfuscation, backdoors, unauthorized persistence, destructive effects and
suspicious/insecure code. Unfamiliar syntax alone proves no maliciousness.

Each judge returns one record/question:
The reviewer returns one record/question:

```text
question_id: <ID>
Expand All @@ -185,21 +183,19 @@ evidence: path:lines; credential source -> sink/destination or relevant control
reason: <one evidence-grounded sentence>
```

UNKNOWN = abstention: missing evidence/unresolved applicability, never success.
UNKNOWN = missing evidence/unresolved applicability, never success.
Absent path: YES only with code-grounded not-applicable explanation.

## Decision

- Compare 2-3 answers/question side by side: model, vote, evidence, disagreement.
Agreement (2/2 or 2/3) = triage signal, not security proof.
- Parent inspects every NO: trace source-to-sink/unsafe effect; resolve through
code evidence or fix. Never outvote credible minority findings. Resolve UNKNOWN
coverage explicitly; third YES cannot supply missing evidence.
- PASS only if every question has two evidence-backed YES votes, all negative
code evidence or fix. Resolve UNKNOWN coverage explicitly with the same reviewer;
an unsupported affirmative answer cannot supply missing evidence.
- PASS only if every question has an evidence-backed YES from the single reviewer, all negative
findings resolved, relevant coverage complete.
- Otherwise block affected execution/upload; continue read-only diagnosis/fixes.
Unresolved suspicion: focused security review, not endless judge debate.
- Report session intent/provenance, base/hashes, models, votes, confirmed findings with paths/lines,
Unresolved suspicion remains blocked; do not retry reviewers or debate indefinitely.
- Report session intent/provenance, base/hashes, reviewer/model, answers, confirmed findings with paths/lines,
unresolved evidence and PASS/BLOCKED/UNAVAILABLE. Store in session artifacts;
never commit raw logs.
- Intent/code/import/dependency/lock changes invalidate affected verdicts. Review new
Expand All @@ -214,13 +210,41 @@ Report this even when nothing was found; incomplete coverage is not a clean bill
of health. Include it in the user handoff, not only in private artifacts or judge
replies, and do not imply that review authorizes execution.

### Annotated Markdown Change Overview

Alongside every review result, show the user an annotated overview of changed
authored `.md` sources, including workflow frontmatter, executable snippets,
prompts, imports, and review guidance. Ignore generated `.lock.yml` files in
this overview; do not reproduce their diffs or annotate generated YAML lines.
Generated locks remain execution-review evidence, not the user-facing source overview.

For each security-relevant source change, provide a revision-pinned Markdown
link and line range, the before/after behavior, and a plain-language annotation
explaining its security impact: credentials and recipients, permissions, output
limits, untrusted-input boundaries, executable provenance, persistence, sandbox
controls or approval requirements. Distinguish improved, unchanged and weakened
controls from unresolved evidence. Explain the practical effect rather than merely
restating the changed text. Redact secret values.

Use a compact table:

| Markdown source | Change | Security impact |
| --- | --- | --- |
| [workflow.md:10–14](<revision-pinned-source-url>) | Custom-token-only launch becomes Actions-token fallback. | Removes the custom-secret dependency; approved routes and privileged-job permission checks remain required. |

If no authored Markdown changed, say so and summarize the relevant code changes
without inventing a source diff. This overview is required even when there are
no findings or the review is blocked; it does not replace the evidence or findings.
Show it in the user response, not only in a private artifact, and do not modify
reviewed workflow Markdown merely to insert annotations.

Example: "Security review passed for the emitted dry-run lock: credential flows,
executable provenance, and sandbox controls were checked, with no confirmed issues."

### Explain Blocked or Unavailable Debugging

A refusal must explain **why** the requested execution or upload is blocked.
"Security review failed", "inconsistent judgments", or "missing coverage" alone
"Security review failed" or "missing coverage" alone
is not an actionable explanation. After the result sentence, identify each
material blocker:

Expand All @@ -243,13 +267,13 @@ Metadata alone does not establish a missing firewall; inspect the actual AWF
configuration and invocation. Distinguish unchanged intentional controls from
new regressions. Apply the [dry-run dangerous-feature filter](#dry-run-dangerous-features)
to authored workflow configuration, not built-in engine implementation flags.
Explain contradictory vote/reason pairs without silently changing votes or
Explain contradictory answer/reason pairs without silently changing answers or
counting them as a passed gate.

When no code defect is confirmed, say so. Link the gate that requires withholding
execution and identify the unresolved question and missing evidence instead of
inventing a problematic source line. For example: "Live dispatch is blocked by
the [two-affirmative-vote requirement](#decision): `secret_disclosure` has unresolved
the [single-reviewer evidence requirement](#decision): `secret_disclosure` has unresolved
review coverage for the [artifact redaction path](<revision-pinned-source-url>).
No credential leak is confirmed. The agent must resolve that source coverage
before dispatch; additional user authorization cannot replace the review."
Expand Down
27 changes: 14 additions & 13 deletions .github/aw/work-queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: Agent instructions for Git-backed work queue producers, dispatchers

`tools.work-queue` schedules immutable DAGs and Claim-scoped effects;
`work-queue.jsonl` is authoritative. V3 has no storage selector; Issues/PRs are
dependencies. Weights distribute Claim opportunities, not CPU/completions.
dependencies.
For issue-backed WorkQueueOps, see [WorkQueueOps](../../docs/src/content/docs/patterns/workqueue-ops.md).

## Select the role
Expand All @@ -17,10 +17,10 @@ For issue-backed WorkQueueOps, see [WorkQueueOps](../../docs/src/content/docs/pa
- **Worker:** require compiler-supplied version-3 `claims`; never override
reserved `work_queue_assignment` or caller context.

Trust compiler/runtime role, not snapshot metadata; don't downgrade workers.
AW authenticates run/workflow/revision; reruns cannot inherit attempt 1.
An absent branch is uninitialized/empty. Its first producer submit atomically
creates it with compiled Policy and Work; reads and dispatch alone never do.
Trust runtime roles, not snapshots; AW authenticates the native run, workflow and
revision; reruns cannot inherit attempt 1.
An absent branch is uninitialized/empty; the first producer submit atomically
creates compiled Policy and Work. Reads and dispatch alone never do.
Reject malformed or unsupported existing logs.

## Configure deployment
Expand All @@ -45,7 +45,6 @@ use one default pool, weighted-priority, concurrency 16, pending 4,096,
singleton assignments and three attempts with 30-second backoff.
First submit automatically installs compiled Policy and Work.
Standalone seeding is unsupported; AW and Actions authorize the trusted submission.
No producer enrollment, principal or credential settings are needed.
Economic Policy updates require quiescence and explicit
`policy --from-config --epoch EPOCH`; config edits never rewrite active state.
Compatible deployments evolve future execution without draining.
Expand Down Expand Up @@ -76,6 +75,10 @@ Upgrade all closed-schema readers first. See the
- Trusted processing refreshes the ledger and commits a compatible fair prefix.
CAS losers discard tentative choices/charges and recompute.

ESLint, daily-report and conformance agents explain staged plans only.
Trusted controls validate the batch; empty dispatcher intents fail.
Unassigned dispatchers cannot emit `noop` or `report_incomplete`.

## Execute original Claims

- Process only the assignment's original Claims and stored plans. Scope every
Expand All @@ -86,16 +89,14 @@ Upgrade all closed-schema readers first. See the
- Finish each member with `work_queue_claim_finish`, using
`outcome: "completed"` or `"cancelled"`. Missing finish authorizes no effects;
valid siblings settle independently.
- Default to one Claim. Enable batching only for compatible Work with substantial
reusable setup; never treat the batch's native conclusion as every Work's Result.
- Default to one Claim; batch only compatible Work with reusable setup. A native
batch conclusion is not every Work's Result.
- Treat finish as an intent: trusted processing publishes Completion before
scoped effects and Result only after independently verified delivery.

Use `<mcp-clis>` subcommands with one JSON argument.
`uninitialized` plus null SHA is empty;
first `work_queue_submit` atomically installs compiled Policy and Work. Submit
before dispatch. Existing policyless ledgers are failures; workers require
Policy. `status: "staged"` is no grant or launch.
Use `<mcp-clis>` with one JSON argument. `uninitialized` plus null SHA is empty;
first `work_queue_submit` installs Policy and Work. Submit before dispatch.
Policyless ledgers fail; workers require Policy; `status: "staged"` is no grant.

## Dependencies and recovery

Expand Down
1 change: 1 addition & 0 deletions .github/skills/agentic-workflows/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Load these files from `github/gh-aw` (they are not available locally).
- `.github/aw/instructions.md`
- `.github/aw/intent.md`
- `.github/aw/jobs.md`
- `.github/aw/ledger.md`
- `.github/aw/linter-workflows.md`
- `.github/aw/llms.md`
- `.github/aw/loop.md`
Expand Down
24 changes: 24 additions & 0 deletions .github/skills/agentic-workflows/tests/test_guidance.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,30 @@ def checklist(text):


class GuidanceTests(unittest.TestCase):
def test_single_security_reviewer_and_markdown_overview(self):
gate = (ROOT / ".github/aw/debug-security-review.md").read_text()
self.assertIn("exactly one read-only security-review agent", gate)
self.assertIn("same reviewer", gate)
self.assertIn("every question has an evidence-backed YES", gate)
self.assertIn("## Single Reviewer", gate)
self.assertIn("### Annotated Markdown Change Overview", gate)
self.assertIn("Ignore generated `.lock.yml` files in", gate)
self.assertIn("Generated locks remain execution-review evidence", gate)
self.assertIn("Show it in the user response", gate)
self.assertIn("| Markdown source | Change | Security impact |", gate)
for obsolete in (
"Launch two independent", "third independent cheap judge",
"two evidence-backed YES votes", "two-affirmative-vote requirement",
):
self.assertNotIn(obsolete, gate)
for skill in ("debugging-workflows", "review-agentic-workflows"):
with self.subTest(skill=skill):
text = (ROOT / f".github/skills/{skill}/SKILL.md").read_text()
self.assertIn("debug-security-review.md#single-reviewer", text)
self.assertIn(
"debug-security-review.md#annotated-markdown-change-overview", text
)

def test_customer_case_remedy_order(self):
case = FIXTURES["model_mismatch"]
self.assertIn("v0.89.21", case["lock_header"])
Expand Down
7 changes: 7 additions & 0 deletions .github/skills/debugging-workflows/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ reference, not a separate execution policy. Respect explicit no-dispatch context
Apply its live-outcome table, credential triage and untrusted-evidence rules.
Without accessible existing logs, use source/fixtures; never dispatch for evidence.

Require one read-only security-review agent per revision, not multiple judges.
Show the user an annotated overview of changed authored `.md` sources explaining
their security impacts; ignore `.lock.yml` diffs in that overview while retaining
generated locks as execution-review evidence. Follow the shared
[single-reviewer](../../aw/debug-security-review.md#single-reviewer) and
[annotated overview](../../aw/debug-security-review.md#annotated-markdown-change-overview) rules.

Workflow registration/activation and secret presence, validity, or expiry are
runtime readiness checks, not pre-dispatch prerequisites. Do not require workflow
or secret inventories or organization-admin metadata access. An otherwise
Expand Down
14 changes: 13 additions & 1 deletion .github/skills/review-agentic-workflows/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,19 @@ Reference workflow authoring skill guidance at: https://raw.githubusercontent.co
## Agent ownership

The agent performs setup, source and runtime-dependency inspection, independent
review coordination when required, compilation/scanners, evidence collection,
single-reviewer coordination, compilation/scanners, evidence collection,
finding resolution and cleanup. Do not give the user a technical preflight
checklist to execute. Report concrete blockers and request only authorization
or decisions the agent cannot supply. For debugging and live-test gates, follow
the [shared security-review guidance](../../aw/debug-security-review.md);
the agent prepares the evidence for human validation rather than asking the
user to conduct the review.

Require exactly one read-only security-review agent per reviewed revision.
Resolve coverage questions with that reviewer; do not require parallel judges,
majority votes or a tie-breaker. Follow the shared
[single-reviewer rules](../../aw/debug-security-review.md#single-reviewer).

The user may explicitly authorize live debugging for the current session rather
than approve each run. Record the grant and agreed scope/bounds; reuse it for
in-scope revisions only after the agent repeats the required technical review.
Expand Down Expand Up @@ -152,6 +157,13 @@ Recommend minimal, safe optimizations that keep or improve security posture.

## Review output contract

Show an annotated overview of changed authored `.md` files, excluding generated
`.lock.yml` diffs from the overview. Link source lines, describe before/after
behavior, and explain the security impact of each relevant change in plain
language. Include preserved controls and unresolved evidence, not just findings.
Keep generated locks in the execution-review evidence. Follow the shared
[annotated overview contract](../../aw/debug-security-review.md#annotated-markdown-change-overview).

Always provide a short user-visible result sentence for each security review and
dry-run attempt, including failed, blocked, or unavailable checks. State the
artifact/scope, outcome, what was actually checked, and any material finding or
Expand Down
Loading
Loading