Skip to content

Enable Atomic built-in tools and schedule the Atomic smoke test - #77020

Merged
pelikhan merged 5 commits into
mainfrom
atomic-tools-schedule-storage
Nov 12, 2026
Merged

pelikhan merged 5 commits into
mainfrom
atomic-tools-schedule-storage

Conversation

@lavaman131

@lavaman131 lavaman131 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #77011. cc @pelikhan

Full Atomic, minus its one interactive tool

gh-aw always runs headless, so the sample now runs Atomic with its whole feature set except ask_user_question:

  • --exclude-tools ask_user_question replaces the read,bash,edit,write,mcp__* allowlist. Every Atomic tool is available (workflows, subagents, Intercom, web tools, todo, native MCP), including tools a repository registers through its own extensions. An exclude list rather than an allowlist is deliberate: a repository adds a custom tool by committing an extension, without editing the engine definition.
  • --approve replaces --no-approve, and --no-skills and --no-prompt-templates are dropped, so the repository's .atomic/ extensions, workflows, skills, prompt templates and settings load, along with the skills gh-aw installs into .atomic/skills.
  • gh-aw's "Restore agent config folders from base branch" step replaces .atomic/ with the base-branch copy before the agent starts, so a pull request cannot supply its own extensions or workflows; only merged ones load.
  • Still off: saved sessions (durable storage comes in a separate opt-in PR), startup network checks, telemetry, and themes (terminal UI only).
  • Headless behavior: a workflow launch returns its run ID, and a durable approval gate stays pending rather than proceeding. Web tools reach only domains the workflow allows.

Smoke schedule and cooldown

  • Smoke Atomic now also runs every two days, as @pelikhan asked (schedule: every 2 days, compiled to 0 0 */2 * *). TestAtomicSmokeContract previously asserted there was no schedule; it now pins the new trigger.
  • The post-merge Smoke Atomic run (43141226682) failed only at install: ETARGET … No matching version found for @bastani/atomic@0.9.79 with a date before 11/9/2026, from the default 3-day npm release-age cooldown. behaviors.installation.cooldown is deprecated and ignored, and strict mode rejects engine.cooldown: 0, so Smoke Atomic and the Atomic conformance workflow set engine.cooldown: 1, the minimum strict mode allows. Their Atomic install now runs with NPM_CONFIG_MIN_RELEASE_AGE: '1'.

Validation

  • go test ./pkg/workflow -run Atomic passes.
  • gh-aw compile smoke-atomic engine-conformance-atomic daily-declarative-engine-improver --validate: 3 succeeded, no deprecation warning.
  • Ran the published 0.9.79 with the final arguments in a repository containing .atomic/extensions/repo-probe.ts (a custom tool), .atomic/workflows/repo-probe-flow.ts and .atomic/skills/repo-probe-skill/SKILL.md, against a local OpenAI-compatible stand-in for the AWF proxy that issued a workflow list call: the model was offered 16 tools including repo_probe and not ask_user_question, the skill was in the system prompt, workflow list returned repo-probe-flow with the built-in workflows, and the run exited 0. Not yet run on Actions.

… its cooldown

- Replace the read,bash,edit,write,mcp__* allowlist with --exclude-tools ask_user_question, so Atomic's default and bundled-extension tools (workflows, subagents, Intercom, web access, native MCP) are available; only the interactive question tool is excluded.
- Run Smoke Atomic every two days as well as on manual dispatch.
- Set engine.cooldown: 1 in the Atomic smoke and conformance workflows (the minimum strict mode allows), so a new pin installs the day after release instead of after three days.
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- --exclude-tools

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the risk, addressed in cbf78863 without the suggested change. Allowing only ask_user_question would remove every tool the agent needs. The exclude list is now an explicit allowlist of exactly the reviewed tools: read,bash,edit,write,find,search,kill,todo,workflow,subagent,intercom,web_search,code_search,fetch_content,get_search_content,mcp__*. Tools added by a later Atomic release stay off until a reviewed bump lists them. Verified against the published 0.9.79: it offers exactly these 15 tools plus MCP, and ask_user_question is not among them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update (cbc7349b): at the maintainer's direction the sample now runs Atomic with its full feature set, minus the interactive ask_user_question, so this is back to --exclude-tools ask_user_question, now with --approve. The exclude list is deliberate: repositories add custom tools and workflows by committing them to .atomic/, which an allowlist in the engine definition would block. .atomic/ is restored from the base branch before the agent starts, so only merged extensions load. Impact stays bounded by the firewall's network.allowed, read-only GitHub MCP, threat detection and safe outputs, and strict mode.

description: Smoke test for Atomic headless execution and native MCP
name: Smoke Atomic
on:
schedule: every 2 days

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schedule was requested by the maintainer, and it doesn't add a new kind of exposure: each scheduled run does exactly what a manual dispatch already does. The containment is deterministic, not prompt text: strict: true, GitHub MCP in read-only mode (GITHUB_READ_ONLY: 1), the AWF firewall with network.allowed limited to defaults, github and go, threat detection on the agent output before anything is published, and safe outputs limited to create_issue and noop. The titles come from PRs merged in this repository, which requires maintainer action. gh-aw's other smoke workflows (for example smoke-agent-all-merged) already run every 2 days under the same model. Keeping the schedule.

List Atomic's default and bundled-extension tools explicitly instead of excluding ask_user_question, so tools added by later Atomic releases stay off until a reviewed bump lists them. The enabled set is unchanged.
Comment thread .github/workflows/shared/atomic.md Outdated
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- read,bash,edit,write,find,search,kill,todo,workflow,subagent,intercom,web_search,code_search,fetch_content,get_search_content,mcp__*

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this set; it is intentional for the Atomic sample. The suggested subset (read,find,search,todo) would break Smoke Atomic itself: step 1 needs mcp__* for native GitHub MCP retrieval, and the build check needs bash. The impact of a steered tool call is bounded by deterministic controls outside the model, not by trimming the list: the AWF firewall limits network egress to the workflow's network.allowed (defaults, github and go here), so the web tools cannot reach arbitrary hosts; GitHub MCP runs read-only (GITHUB_READ_ONLY: 1); workflow and subagent run inside the same sandbox with the same tool policy; nothing leaves the run except safe outputs (create_issue, noop), which threat detection reviews before publication; and strict mode applies. The allowlist is explicit, so tools from later Atomic releases stay off until a reviewed bump lists them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update (cbc7349b): at the maintainer's direction the sample now runs Atomic with its full feature set, minus the interactive ask_user_question, so this is back to --exclude-tools ask_user_question, now with --approve. The exclude list is deliberate: repositories add custom tools and workflows by committing them to .atomic/, which an allowlist in the engine definition would block. .atomic/ is restored from the base branch before the agent starts, so only merged extensions load. Impact stays bounded by the firewall's network.allowed, read-only GitHub MCP, threat detection and safe outputs, and strict mode.

@lavaman131
lavaman131 marked this pull request as ready for review November 12, 2026 02:56
Copilot AI dynamic review requested due to automatic review settings November 12, 2026 02:56
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~2 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~3 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #77020

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Matt Pocock Skills Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Matt Pocock Skills Reviewer · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K · ◷

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #77020

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-12T03:00:02Z
review_event: REQUEST_CHANGES
top_themes: [supply-chain cooldown reduction on scheduled workflow]
files_reviewed: [.github/workflows/engine-conformance-atomic.md, .github/workflows/shared/atomic.md, .github/workflows/smoke-atomic.md, pkg/workflow/atomic_workflow_test.go]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~2 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit tool allowlist and scheduled trigger are documented and consistent with the stated Atomic smoke-test goals. However, the new one-day npm cooldown weakens supply-chain protection for an unattended, recurring workflow with copilot-requests: write; please preserve the stronger default gate or pin a reviewed package version.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~2 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review to run again

engine:
id: atomic
# Validate a newly pinned Atomic release the day after it ships; strict mode allows at least one day.
cooldown: 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowering the npm release-age guard to one day for this unattended scheduled workflow materially increases supply-chain exposure because a newly published compromised Atomic package can be installed before the normal three-day delay while the job has copilot-requests: write.

💡 Keep the stronger install gate

The workaround addresses a transient ETARGET failure, but it trades away protection on a recurring job with write-capable permissions. Prefer pinning a reviewed Atomic version (or otherwise preserving the default cooldown) and update the pin through the normal review process.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c0e994bc: Atomic keeps cooldown: 1, while the threat-detection Copilot engine has an explicit cooldown: 3 override.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /codebase-design, and /tdd. No blocking defects found. Three follow-ups, inline.

📋 Key Themes & Highlights

Key Themes

  • Unscoped cooldown side effect: engine.cooldown: 1 also lowers the detection job's Copilot install from 3 to 1 days.
  • Duplicated allowlist: the Atomic tests still hard-code the old tool list.
  • Brittle schedule assertion: the test pins source text rather than the compiled cron.

Positive Highlights

  • ✅ The lock files regenerate byte-for-byte from HEAD, so the committed artifacts match the sources.
  • ✅ The allowlist explicitly excludes ask_user_question, which is the right call for headless runs.
  • ✅ The shared/atomic.md doc comment now matches the allowlist.

The GHAS threads on the broad mcp__* allowlist and the scheduled prompt-injection surface are already covered by the author's replies, so I have not repeated them.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

engine:
id: atomic
# Validate a newly pinned Atomic release the day after it ships; strict mode allows at least one day.
cooldown: 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] engine.cooldown: 1 is applied globally, not only to Atomic. The detection job in the compiled lock also installs Copilot with this cooldown, so its --cooldown-days drops from 3 to 1 (smoke-atomic.lock.yml line 2157). That weakens the supply-chain delay for the detection job, and the PR does not mention it. Confirm this is intended, or scope the setting to Atomic only.

💡 Evidence

Compiling HEAD with the cooldown line removed yields --cooldown-days 3 in the detection step and NPM_CONFIG_MIN_RELEASE_AGE: '3' for Atomic. Compiling HEAD as-is yields --cooldown-days 1 in the detection step. The detection job uses detection-engine: copilot from shared/atomic.md.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c0e994bc: the Atomic install retains its one-day age, and the detector now has a separate three-day cooldown override.

Comment thread .github/workflows/shared/atomic.md Outdated
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- read,bash,edit,write,find,search,kill,todo,workflow,subagent,intercom,web_search,code_search,fetch_content,get_search_content,mcp__*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The allowlist now lives in shared/atomic.md, but the Atomic tests still hard-code the old read,bash,edit,write,mcp__* list. Those tests pass because they supply their own argument lists, so nothing checks the real allowlist.

💡 Suggested test

Add a test that reads the --tools value from shared/atomic.md and asserts it matches the documented set. It should also assert that ask_user_question is absent, as the doc comment states. Stale copies are at pkg/workflow/atomic_environment_test.go lines 47 and 110 and pkg/workflow/atomic_output_redaction_test.go lines 72 and 86. A single source of truth keeps these in sync.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c0e994bc: the test pins the exact parsed execution args and rendered exclude flag; the harness fixtures now derive their args from the shared Atomic definition instead of carrying stale copies.

Comment thread pkg/workflow/atomic_workflow_test.go Outdated
require.NoError(t, err)
assert.Contains(t, string(source), "shared/atomic.md")
assert.NotContains(t, string(source), "schedule:")
assert.Contains(t, string(source), "on:\n schedule: every 2 days\n workflow_dispatch:\n")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] This assertion matches exact source text, including blank lines and indentation, so harmless reformatting of the frontmatter breaks it. It also does not check the compiled cron. Assert the parsed schedule instead, or check that the lock file contains 0 0 */2 * *.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c0e994bc: the smoke contract now asserts the generated lock's 0 0 */2 * * cron instead of matching exact frontmatter formatting.

Return to --exclude-tools ask_user_question so every Atomic tool is available, including tools a repository adds through its own extensions. Replace --no-approve with --approve and drop --no-skills and --no-prompt-templates, so the repository's .atomic/ extensions, workflows, skills, prompt templates and settings load, along with the skills gh-aw installs into .atomic/skills. gh-aw restores .atomic/ from the base branch before the agent starts, so a pull request cannot supply its own.
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- --approve

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Atomic still enables native web fetching and search when workflows explicitly disable them, leaving those restrictions unenforced.

4 open issues
What changed in this PR

Extends the experimental Atomic engine sample following the 0.9.79 integration and adds recurring smoke validation.

Changes:

  • Expands the explicit native-tool allowlist while excluding interactive questions.
  • Schedules Smoke Atomic every two days and sets a one-day install cooldown for smoke and conformance workflows.
  • Updates generated workflows, schedule assertions, and documentation of headless behavior.
File Description
pkg/​workflow/​atomic_workflow_test.go Updates the smoke contract to require the new schedule.
.github/​workflows/​smoke-atomic.md Adds the recurring schedule and one-day cooldown.
.github/​workflows/​smoke-atomic.lock.yml Regenerates scheduling, tool selection, cooldowns, and metadata.
.github/​workflows/​shared/​atomic.md Expands the tool allowlist and documents headless behavior and limits.
.github/​workflows/​engine-conformance-atomic.md Sets the Atomic conformance install cooldown to one day.
.github/​workflows/​engine-conformance-atomic.lock.yml Regenerates tool selection, cooldown, and metadata.
.changeset/​patch-atomic-builtin-tools-smoke-schedule.md Records the sample changes in a patch release note.

Review effort: Max (auto)

Comment thread .github/workflows/shared/atomic.md Outdated
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- read,bash,edit,write,find,search,kill,todo,workflow,subagent,intercom,web_search,code_search,fetch_content,get_search_content,mcp__*
Comment thread .github/workflows/shared/atomic.md Outdated
- --no-approve
- --tools
- read,bash,edit,write,mcp__*
- read,bash,edit,write,find,search,kill,todo,workflow,subagent,intercom,web_search,code_search,fetch_content,get_search_content,mcp__*
engine:
id: atomic
# Validate a newly pinned Atomic release the day after it ships; strict mode allows at least one day.
cooldown: 1
Comment thread .github/workflows/shared/atomic.md Outdated
tools are enabled inside the gh-aw sandbox. This sample does not advertise
bundled extensions remain loaded. An explicit allowlist enables Atomic's default
and bundled-extension tools inside the gh-aw sandbox: read, bash, edit, write,
find, search, kill, todo, workflow, subagent, intercom, web_search, code_search,
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (.github/workflows/smoke-atomic.md:16): address unresolved feedback - Review comment
  3. Review (.github/workflows/smoke-atomic.md:16): address unresolved feedback - Review comment
  4. Review (.github/workflows/shared/atomic.md:57): address unresolved feedback - Review comment
  5. Review (pkg/workflow/atomic_workflow_test.go:397): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "cbc7349b64fdb1d6bd4d1e6e4a6302ef6905ffa0",
  "kind": "requested",
  "schema_version": "1",
  "state": "b9c6f4d32af83056884c3982ed6feb94a962c47837678ea6ade5d5bf6405d3cf",
  "work": [
    "43fb2fab4792659ce8a9a5979d64e8fe6e6a43f3219b7bf3a0fabc4ac29749da",
    "a78befba6a8b7b6b0660b3a9ca59aff1129d1db3007f8f15c90b659e5242d9c0",
    "a9ad9ae28dd89de35bca3e9353f293e215067bdc97e069540bcaec8bd06d9005",
    "aeb0e4d812c55b9adeaa6ca02542f622f4350409c2cdbc19cbf26614539e6251"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

…-storage

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan
pelikhan merged commit 38add47 into main Nov 12, 2026
83 checks passed
@pelikhan
pelikhan deleted the atomic-tools-schedule-storage branch November 12, 2026 04:33
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants