diff --git a/docs/architecture/adr/0013-factory-mvp-reference-architecture.md b/docs/architecture/adr/0013-factory-mvp-reference-architecture.md new file mode 100644 index 00000000..4714a4c8 --- /dev/null +++ b/docs/architecture/adr/0013-factory-mvp-reference-architecture.md @@ -0,0 +1,698 @@ +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# 0013. Factory MVP reference architecture: a declared control plane, one run contract, one release + +## Status + +proposed + +## Context + +The agentic factory in this repo grew by emergent design and is now treated as a proof of concept. +`docs/prd/agentic-software-factory.md` states what a production MVP must do. The review in +`docs/architecture/factory-mvp-architecture-review.md` measured the POC against it at commit +`aef61164` (2026-10-10) and analysed the options. This record states the decision; the options are +in the review, §§4–11, and are not repeated here. + +The review found seven structural causes behind eleven recurring failure classes. Each is a +decision the POC never made once, so it was made many times, locally: + +1. **Work-item state is implicit.** No file declares states or transitions. State is recomputed by + each script from labels, about eighty HTML-comment marker shapes, review objects and checks. The + only marker registry is one regex (`scripts/lib/maintainer-comment-gate.sh:74`). +2. **Work is carried by events and concurrency slots.** A request exists only as a queued run, and + the platform drops both runs and scheduled ticks (`.github/workflows/dev-lead-reusable.yml:76-85`; + `.github/workflows/pr-review-sweep.yml:15-18`). Fifteen workflows exist to find and restart what + fell through. Nothing admits runs in one place. +3. **The model holds the write credential and reports on itself.** In the two busiest roles the + harness polices claims afterwards (`.github/workflows/dev-lead-reusable.yml:173`; + `scripts/lib/addressed-claim-verify.sh`). +4. **Agents act as the Maintainer.** One account is the human, dev-lead and eight personas + (ADR-0008). The loop ceiling keys on login and so cannot count them + (`scripts/lib/pr-automation-budget.sh:54-59`), the Maintainer cannot review their pull requests, + and the fleet and every session draw on one API quota. +5. **There is no single unit of release.** Changes to dev-lead and pr-review are inert until a tag + moves (`AGENTS.md:25-33`); persona prompts and manifests are live at merge + (`.github/workflows/persona-runner.yml:23-24,39`). One fact is written in up to eight places + across two repos. +6. **An approval is bound to a commit hash**, can be withheld or dismissed by several independent + gates, and one merge forces work on every open pull request + (`scripts/review-one-pr.sh:838-842`; `.github/workflows/auto-rebase.yml:25-30`). +7. **Verification does not bind.** The `main` ruleset requires five checks (read live, + 2026-10-10); the test suite, the validators and the evals are not among them. Decision logic sits in shell and inline workflow + steps that cannot run on the maintainer's machine (`AGENTS.md:174-181`). + +Three things the earlier records decided bear directly on this one. + +**ADR-0007** weighed and rejected an org-level GitHub App as *ingress* and a webhook-native runtime, +"at the current fleet size", because both need a hosted endpoint, a second secret store and their +own deduplication. That reasoning stands. This record uses a GitHub App as an *identity* whose +tokens are minted inside Actions. It hosts nothing and does not reopen ADR-0007. + +**ADR-0008 and interaction model §12.2** put the personas and dev-lead on the owner account. The +review engages with their reasoning point by point (review §6.4). Their stated objection to a +distinct identity was a new machine user with a new long-lived token. A GitHub App has neither, and +GitHub's terms permit only one free machine account, which the org already uses for the reviewer. + +**ADR-0010** recorded that ADR-0000 has no status for an ADR that extends another. That gap is felt +here and is not fixed here. + +One platform fact changed during the review. A concurrency group's pending depth of one is now a +default: GitHub documents `concurrency.queue: max`, up to 100 pending runs. It is a useful +mitigation for the POC. It does not change this decision, because a deeper queue still loses a +request when a tick is dropped, a dispatch fails or a run is killed. + +**This decision was attacked before it was proposed.** The review compares it with seven other +systems (§13) and records an independent adversarial pass (§14). That pass found the first draft +sound in four decisions and wrong in three ways: it committed to two mechanisms before measuring +them, it removed brakes the POC has, and it had no stopping rule. The boundaries below include the +changes it required. It also found two ceilings this decision does not raise, which are stated in +Consequences. + +**Where it is built was decided on 2026-10-10.** The first draft kept the factory as a tree inside +this repo and left a separate repo to be decided at the end. The review weighed building in place +against building greenfield and switching over (§18). In place, every factory change pays the +POC's price: about 56 CI jobs a push, nine reviewers, and an approval gate that issues for no open +pull request. A literal switch-over is not available either: GitHub Actions does not redirect +reusable-workflow references, so renaming or replacing this repo would break every caller at once. +The maintainer chose the hybrid: build in a new repository, cut over through the channel tags this +repo already has. That lifts the "no new repos" half of the constraint set in June 2026. The "no +always-on infrastructure" half stands. + +In this record "the factory repo" is the new repository, `petry-projects/software-factory` (named +by the maintainer on 2026-10-10), and "this repo" is `petry-projects/.github-private`, which hosts +the POC. + +**A note on the number.** This record was drafted and discussed as ADR-0012. While it was in review, +ADR-0012 was assigned on `main` to the maintainer comment gate's exemption set, so this record +takes the next number. Earlier links and comments that say "ADR-0012" about the factory MVP mean +this record. + +**ADR-0012 (the comment gate's exemption set, proposed)** narrows which pull-request comments the +POC's gate counts, by registered marker or one exact body form. It governs the POC. Decisions 7 and +12 below replace that gate's premise, that a comment body is what gates, and the merge-integrity +follow-on will supersede ADR-0012 when the factory's gate takes over. + +**Scope and precedence.** This record fixes the structure every new factory component follows. It +supersedes no ADR by itself. An accepted ADR keeps governing the POC code path it describes until a +follow-on ADR, named below, supersedes it in the same change that makes the new behaviour true in +the running system. Flipping statuses on acceptance would make the log claim what the system does +not yet do, which is the defect ADR-0008 was written to repair, in reverse. + +## Decision + +We will rebuild the factory's core as **a declared control plane, one run contract and one release**, +on GitHub Actions, with no always-on component. We will build it in a repository of its own, beside +the POC, and cut over one +repo and one intent family at a time. Terms are those of the PRD glossary. + +### 1. One self-contained repository, with decision logic that does no I/O + +We will build the factory in a new repository of its own: Lifecycle, Role manifests, schemas, +configuration, Kernel, adapters, Harness, Controller, release tooling and their tests. The Kernel is +Python 3 using the standard library only. All I/O is in adapters. Shell is step glue. + +The boundaries a check can assert: + +- **No module under `kernel/` performs I/O.** It imports nothing that opens a file, a process, a + socket or the environment, and it takes the clock as an argument. +- **The factory repo reads nothing from this repo at run time.** Code carried over from the POC is + copied in with its tests and owned there. Nothing is sourced, fetched or checked out across the + two. +- **No decision logic in workflow YAML.** A `run:` block in a factory workflow is at most ten lines + and calls a module entry point. +- **The factory repo starts as it means to go on.** From its first commit it has the required check + of decision 8, the agent identity of decision 5 as the author of agent changes, and + environment-scoped secrets. + +### 2. A declared Lifecycle, with State derived and never stored + +We will define one Lifecycle per work-item type as versioned data under `lifecycle/`, and compute +each work item's State at every evaluation from what GitHub holds. We keep no second authoritative +store. The work item shows a projection of its State. Labels are the human's input vocabulary. +HTML-comment markers are not state. + +The first draft kept a State record per work item and an append-only Ledger in a git store, as a +provisional choice. The maintainer replaced it on 2026-10-10: most of what that store held is +already on the work item, and a stored copy can only go stale. In this record "the Ledger" now +means the factory's run records and artifacts in GitHub Actions, as indexed in the control room. + +- **State is computed, never stored.** One Kernel function derives it from facts and the factory's + run records, both held by GitHub. +- **Every transition the factory performs for a Release is declared in that Release's Lifecycle.** + An undeclared one fails a check and is reported. +- **No module under `kernel/` takes a comment body as input.** Text written by third + parties becomes typed records in adapters and Roles that are enumerated in one list. The entries + in that list that read legacy markers may only shrink. +- **What only the factory knows is written once, where GitHub keeps it.** That a Run happened and + how it ended is its Actions run record, named for its work item. What the Run cost and what a + model judged are in the Run's artifact, which is immutable and has a digest. Nothing is handed + from job to job through a log. +- **Reviews, checks, labels, merges and thread resolution are read as facts at each evaluation** + and are never trusted from a copy. +- **Facts say whether they are complete.** A list cut off at a page limit, or a truncated diff, is + unreadable. Unreadable facts never converge and never admit. +- **Every pass records the facts it read and the decisions it made,** as an artifact, so that any + decision can be replayed. +- **The control room is a materialized view.** `petry-projects/software-factory-control-room` holds + an index of the run records and artifacts and every item's computed State. Only the Controller + writes it. It is public. For the 90 days GitHub keeps run records it can be deleted and rebuilt; + it may keep what it has collected for longer, for reporting only. A view + rebuilt from GitHub equals the view updated pass by pass, and a check asserts it. +- **No decision reads the control room.** The same facts and run records give the same decision + whatever the view holds, so the view can be wrong or missing without the factory misbehaving. +- **The projection on a work item is rewritten only when the decision changes,** and at most once + per item per stated interval. It contains no value that changes when the decision does not. + +### 3. Level-triggered Admission in one place + +We will admit every Run through one Controller that evaluates current facts and the factory's own +records. The Controller is a poller: one pass evaluates every open work item in every repo. An +event carries no work. It may only ask for a pass now, and any number of such nudges during a pass +collapse into one more pass. The Controller runs in the factory repo. A Run is its own workflow +run, one lane per work item, held from `run` to applied Effects. The first draft woke the +Controller once per event; the maintainer replaced that on 2026-10-10 (decision 11). + +- **A nudge carries nothing.** The Kernel's admission function takes no event payload. +- **When a work item needs the maintainer, the factory assigns it to them or requests their + review,** so that GitHub's own notifications carry it. A fault of the whole factory goes to one + tracking issue. +- **A work item has a class:** agent, maintainer, dependency or outside. An outside item gets no + Run until the maintainer admits it (PRD FR-63), and admission lapses when its author pushes + again. +- **Attempts are counted since the last action on the work item by a person with write access.** + That grants a fresh allowance. No agent's action does, and nor does a stranger's comment. +- **When more items need the maintainer than a declared number, only Runs that unblock a merge are + admitted.** The starting number is 10. +- **A run record counts only if it is the factory's own workflow, on the default branch, started by + a schedule or a dispatch.** Its name identifies nothing. A fork's workflow run is ignored. +- **A Run is started only from a decision a pass recorded.** Only the poller can be started from + outside the factory, and it takes no inputs, so a credential that can start workflows cannot + direct a Run. +- **A fault in the environment is not an attempt.** A failed Run is classed as a fault of the item, + of capacity or of the environment, or as superseded. Only the first counts against the item. Several environment + faults in a row pause Admission, raise one alert, and clear when a probe Run succeeds. +- **A hosted relay supplies low latency.** One small stateless function outside GitHub receives the + Controller App's webhooks for every repo and runs a timer, and turns each into a nudge. It holds + no credential over the factory repo, keeps no state and decides nothing. Correctness never + depends on it. It is the one permitted exception to "no always-on component". +- **One workflow, the one that carries a Run, is the only one that invokes an Engine or references + an LLM credential.** The poller invokes none. +- **At most one Run that can produce Effects is active per work item,** by the lane. Org-wide + limits are decided inside the one pass, which is the only decider, so there is no race to lose. +- **Passes do not overlap, and at most one waits.** A pass starts its successor when something is + due before the next scheduled tick. +- **Hold, budget, duplication and concurrency are evaluated in the admission function and nowhere + else.** A human Pause is a variable. An automatic pause is derived at each pass from recent + faults, so automation cannot clear the human one. +- **A missing or failed capacity reading admits nothing.** It yields a deferral with a reason. + +These brakes sit outside the Kernel, so that a defect in the Kernel cannot disable them: + +- **A human Pause holds when factory code is broken.** The entry shim's caller jobs are conditioned + on a pause variable that the platform evaluates before any factory code runs. +- **A ceiling on admitted Runs, per work item and per org in a rolling window, is enforced outside + `kernel/`,** from a count the Kernel does not write, with its own tests. +- **The count of automated actions since the last human decision is computed from GitHub's record + of actors,** not from the Ledger. +- **On a metered credential, a monthly cap in currency stops Admission.** It is read from the + provider's own usage figure. On the subscription there is no currency cap, and the Run ceiling + is the brake. + +And a nudge is bounded in what it can cost: + +- **A consumer repo carries no factory file.** The poller reads it and the relay hears its events. + +### 4. One run contract; a job is a credential boundary + +We will give every Role one contract: a schema-versioned Task in, a schema-versioned Result out, +with every side effect expressed as a requested Effect. Three jobs carry it: `run`, `verify`, +`apply`. + +- **No job holds both an LLM credential and a credential that can write to GitHub.** +- **No job that holds a write credential executes code from the work item's branch.** Only `verify` + does, and `verify` holds no secret. +- **A Result that fails schema validation applies no Effect.** +- **A statement that names a commit is written by `apply` from a commit it pushed and read back.** + The model's output schema has no field for a commit hash. +- **A Role's permitted Effects are listed in its manifest and enforced by `apply`.** This is how + ADR-0005's write opt-in is enforced. +- **`run` and `verify` declare `permissions: {}` and neither restores nor saves a cache.** `permissions: {}` + restricts only `GITHUB_TOKEN`; the Actions cache credential is separate. So these jobs use no cache + action, run on ephemeral runners, and run in a context that cannot write the default-branch cache + scope later jobs restore. A lint asserts all three. +- **The checks that judge a model's diff run in a job that executes none of it,** and `apply` + re-derives the digest of what it pushes instead of trusting an earlier job. +- **`apply` refuses, to a Hold, any change under `.github/`, to `CODEOWNERS`, or to the files that + define a consumer's setup and test commands,** unless the Role's manifest permits that Effect and + a human transition is recorded. +- **Every Effect type declares a recovery key that is not text in a body.** For an Effect with no + natural idempotency key, the Result artifact is the intent record: it exists before `apply` + starts. If it is missing or its digest does not match, nothing is applied. +- **A Run that ends with no Result is still a run record.** It counts as an attempt unless its fault is classed + as capacity or environment, and the Lifecycle declares what follows it. +- **A Run that reads text from an outsider or a third-party reviewer has no shell and no network + tool.** +- **An approval is submitted for exactly the commit that was admitted,** and the commit bundle is + taken by the identifier and digest the `run` job reported. +- **`apply` re-reads the work item before any Effect,** and writes nothing if the item is held, + paused, closed or no longer the content the Run was admitted for. +- **`verify` never replaces the consumer's own required checks.** Its result counts in place of the + author's, and the consumer's checks on what was pushed remain the verdict. +- **Feedback moves left; the verdict does not.** `run` executes the pre-flight checks and returns + their Findings to the author for a bounded number of repair turns. `verify` executes the same + checks, at the same pinned versions and configuration, and only its result counts toward a Gate. + The checks are defined once and are not a git hook, which the author could edit or skip. + +### 5. Agents have their own identity + +We will have every agent write through a factory GitHub App of its own, one App per Role, with a +token minted per Run and narrowed to the Effects that Run requests, and have approving reviews +submitted by the existing machine user. Code-owner review stays required. The owner account is the +human's alone. + +The maintainer decided on 2026-10-10 that no agent runs as the owner account, and that each Role +has its own App, so that a Role's permissions are a ceiling the platform enforces and not only a +rule in the factory's own token-minting code. The review had recommended one App (§6.4, option c); +this is its option d. The Maintainer may still author changes to the factory repo under the owner +account, as a person. "The App" in the boundaries below means the App of the Role that acts. + +- **The factory's Apps share the name prefix `combsmith-sf-`,** and the consumer file drops + factory-authored events by that prefix. Adding a Role changes no consumer file. +- **Registering an App is a manual step.** A new Role is one pull request and one registration. +- **Every write by a factory identity is reconciled with a recorded Effect.** A write with no match + raises an alert: a credential is in use outside the factory. +- **The model credential is the factory's alone,** not the maintainer's own subscription. If a + subscription cannot be used, the fallback is a metered API credential under a monthly cap. +- **Apps are installed across the org, and each token is minted for one repo.** +- **The machine user approves and does nothing else.** It never pushes and never opens a pull + request, because GitHub does not count an approval from an author or a last pusher. +- **The Maintainer is the only code owner for the factory's own rules** in the factory repo: the + Lifecycle, the authority table, the gate function, the brakes, the invariants and `.github/`. The factory cannot + approve a change to what governs it, whichever Release is serving. +- **Reading is shared; writing is not.** Every `run` job reads GitHub through one reader App that + can write nothing. A Role is given its own App when it first needs to write, so a Role in shadow + has none. + +- **No factory workflow references the owner account's token.** +- **Every Effect in the Ledger has a factory App or the reviewer as its actor.** +- **For every agent-authored pull request, author and approver differ, and the App is not a code + owner.** +- **The count of automated actions since the last human decision does not consult a login list.** +- **No factory secret is readable by a workflow that runs on a work-item branch.** The App key, the + reviewer's token and the LLM credential are environment secrets restricted to the default branch. + A test pull request proves this before any agent uses the App. + +### 6. One Release, one path + +We will release the factory as one line: immutable `factory/vX.Y.Z` tags and moving +`factory/v-next` and `factory/v-stable` channels, on ADR-0002's mechanism. Workflows, +Kernel, Harness, prompts, manifests, Lifecycle and configuration all ride it. Ring membership is one +file in the Release. The factory repo is served by `next`, and `stable` serves consumers. + +- **Every Run records the Release it ran.** +- **Inside a Release, factory files are read at `github.job_workflow_sha`,** never at a branch. +- **The only factory files that run from the default branch are the entry shim in the factory repo + and a workflow in the factory repo that nudges the poller on the factory's own events.** Both are byte-frozen thin callers under ADR-0001: + every job `uses:` a pinned factory reusable and declares no `steps:` or `run:`. The shim has one + static caller job per channel and selects between them with a job-level `if:` on the output of a + pinned ring-resolution job and on the pause variable. Those two non-event predicates are the only + additions to ADR-0007's filter rule. +- **A consumer repo carries no factory file.** Nothing of the factory runs from a consumer's + branches. +- **Rollback executes no file from the Release being rolled back.** +- **A candidate reaches `stable` only when the Ledger holds, for that Release, an eval result for + each production Intent whose inputs changed and verified Effects from the `next` ring.** A run + conclusion is not evidence, and a ring with no Runs of the candidate is not evidence. +- **An Intent is cut over only after an eval set exists for it that exercises the production + path:** the approving Intent before review moves, the code-writing Intents before they move. + +### 7. An approval is bound to content + +We will keep GitHub's dismissal of stale approvals, bind each verdict to a content identity (a hash +of the pull request's own diff against its merge base), and decide approval in one Kernel function. + +- **A verdict is re-issued without a Run only when the content identity equals the one reviewed.** + Any other case is a review. +- **An approval is withdrawn only by the platform on new content or by a recorded Kernel transition + that cites a new unresolved finding,** and that transition also removes the pull request from the + merge queue. No Role dismisses its own approval. +- **The factory updates a pull request's branch only when it conflicts with its base.** +- **A merge without a standing independent approval has an Override record.** +- **An approval is decided from Findings and human transitions** (decision 12). A review thread, or a + review that requests changes, from a person with write access is a Finding from a human Lens and + always gates. A plain comment steers: it reaches the next Run's Task and is acknowledged, and it + gates nothing. A thread from anyone without write access is handled as a third-party reviewer's + would be. This replaces the POC's rule that every comment body needs a verified disposition. +- **A human resolving a review thread in GitHub's interface is a declared human transition.** +- **A Finding is decided by the Role that owns its category,** under an authority table held as + data. The author proposes a Disposition and the owner accepts or rejects it. An owner resolving + a thread it has accepted is a declared Role transition. No Role decides on its own work, and a + security Finding is never declined without an Override. +- **The reviewer resolves every third-party thread it does not adopt.** The ruleset's requirement + that threads be resolved stays in force, and each such thread gets one visible outcome. +- **A domain Role starts in shadow.** It decides on a sample, its decisions are recorded and bind + nothing, and the reviewer owns the category meanwhile. It is promoted to a gate on an eval set + and a measured record, one Role at a time. + +### 8. Verification binds + +We will make the factory's own verification a required check from the first commit of the tree. + +- **`factory / verify` is a required check on the factory repo's protected branch.** It runs on + every pull request and on `merge_group`, with no path filter. +- **A required check is hermetic:** it reads nothing outside the merge commit. +- **A required check is deterministic.** Randomised exploration runs on a schedule and blocks + nothing. +- **The maintainer's own approval never depends on the factory running.** It is the way out when the + Release that serves the factory is broken. +- **The declared Lifecycle is checked mechanically** for termination, undeclared cycles, and Hold + and Pause semantics. + +### 9. One fact, one place + +- **The set of Roles is the listing of `roles/`.** The consumers, each consumer's ring and each + Role's triggers are each written once in the factory repo. +- **Any table of these in a document is generated, and a stale one fails `factory / verify`.** + +### 10. Migration: built greenfield, cut over as a strangler + +- **Factory code is written in the factory repo.** This repo receives defect fixes, configuration + fixes and the do-nothing commits that cutover needs, and nothing else. +- **Cutover is pause, then repoint.** With a poller there is nothing in the factory for a consumer's + workflow to call. First, this repo's channel tag for an intent family moves onto a commit where + that family does nothing. Then the factory's ring file lists the repo for that family, and takes + effect at the factory's next Release. Between the two, nobody serves that family on that repo; + both serving it is what never happens. The maintainer accepted the gap on 2026-10-10. Reversing + the two switches is the rollback. No consumer file changes during cutover. +- **This repo is not renamed, and its name is not reused, while any caller references it.** GitHub + Actions does not redirect reusable-workflow references. +- **Consumers are re-pinned to the factory repo last,** through the standards sweep, after every + intent family is served by the factory at `stable`. This repo then keeps only what GitHub's + `.github-private` convention gives it. +- **For any repo and intent family, exactly one line serves it,** the POC or the factory, as + recorded in the ring file. A check derives this and fails on dual serving. +- **The first factory component is the review intent, running end to end on the factory repo + itself,** so that the factory reviews everything built after it. Until the first Release on `next` + serves the factory repo, the Maintainer approves factory changes. This replaces the first draft's read-only + observer as the first component (maintainer, 2026-10-10). +- **An intent family is cut over for a consumer only after it has run at `next` on the factory + repo, its Release has been promoted to `stable`, and its eval set exists** (decision 6). The comparison of the factory's decisions with the + POC's on live traffic is still made and reported. It is a measure, not a gate. +- **From that point the POC takes defect fixes only.** +- **A compensating workflow is retired in the change that makes it unnecessary, or keeps a recorded + reason to remain.** The number of compensating workflows does not rise during the migration. +- **The migration has a defect policy and a stopping rule, and no dated limit.** The first draft + required a dated limit, a defect policy with a weekly cap and a kill criterion before this + decision was in force. The maintainer settled all three on 2026-10-10: + - **Limit.** None. No period is set. + - **Defect policy.** POC defects are fixed in the POC as needed. There is no weekly cap. + - **Stopping rule.** If the factory is not adopted and used, the maintainer decides to stop. No + measure stops it automatically. The control room shows adoption, use and the factory's results + against the POC's baseline so that the decision is informed. If the migration stops, the + identity change and the configuration fixes are kept and unused factory code is deleted. + - **Decision points.** At the first Release on `next` that serves the factory repo, and at the + first consumer repo served, the maintainer records continue or stop. Before the first, the control room shows how many factory + pull requests the maintainer has approved, against a starting figure of 40. +- **Escaped defects (PRD SM-C1) are baselined before any step that raises the autonomy rate.** +- **Three assumptions are settled on a canary repo before any cutover:** that a pull request + authored by the App needs no extra approval under the live ruleset, which sets + `require_extra_approval_for_unattributed_changes`; that the App can enable auto-merge and enter + the merge queue; and whether sending a nudge counts against GitHub's content-creation limit. + +### 11. The poller and the view are measured + +The first draft adopted a git-backed store and a wake per event provisionally, because every +comparable orchestrator polls from one standing process and keeps control state it can rebuild +(review §13). The maintainer settled both on 2026-10-10: state is derived (decision 2) and the +Controller polls (decision 3). The thresholds written for the provisional choices are kept as +measures of the settled ones. + +- **Over two weeks of serving the factory repo:** handling passes and nudges uses at most 10% of + the org's job-minutes; a pass costs at most 3,000 GraphQL points an hour at peak; and the + 95th-percentile delay from a change that produces no event to a decision is at most 15 minutes. +- **If the delay threshold fails with the relay in place,** + a follow-on ADR decides between a longer-lived polling job and a wake per event. +- **If the view cannot be kept equal to a rebuild, a follow-on ADR replaces it.** Decisions 1 and + 4 to 10 stand either way, because the Kernel does no I/O. + +### 12. Lenses report through one contract; convergence is decided, not discussed + +The maintainer's working hypothesis is that a council of independent Lenses (models, tools and +perspectives) produces a better outcome than one agent alone, across the whole lifecycle. The POC +ran such a council through free-text discussion. The research supports the hypothesis under +conditions (review §15), and those conditions are the boundaries here. + +We will treat every source of judgement on a work item, whether a Role, a third-party reviewer, a +tool or a human, as a Lens that reports Findings in one schema, and we will decide Convergence in +the Kernel. + +- **A Finding is a structured record:** its source, an identity that is not text, a location, a + severity and the content version it applies to. Whether it is open is read from its surface. +- **A third-party reviewer is integrated through a declared contract.** Its entry in the reviewer + registry names the structured surface its Findings are read from: a check run and its + annotations, code-scanning alerts, or review threads. A comment body is presentation for people. + Nothing parses one to decide, waits for one, or is gated by one. A reviewer with no structured + surface gates nothing. +- **A third-party Finding whose category and severity are prose is typed by adoption.** The + in-house reviewer is given it as input and, if it agrees, raises a Finding of its own that + carries those fields. No code parses the prose, and no separate Role exists to type it. +- **What people read is rendered from the records** by the Harness. No Role writes prose that + another component must interpret. +- **Lenses do not address each other.** No Lens's output is another Lens's trigger. Each reports to + the Controller, and none sees another's verdict on a content version before giving its own. +- **One author per change.** Lenses judge a change; they do not co-write it. The acceptance contract + is the first commit on the work's branch, by its own author, and the implementer cannot touch + it: no file in a pull request has two authors. +- **Convergence is a Kernel function:** every Lens required for the item's class and stage has + reported on the current content version, and no blocking Finding is open. An item that has not + converged within its declared bound is placed on Hold. +- **A reviewing Role declares a rubric.** Its criteria are data, each Finding cites one, and a + Result reports on every criterion that applies. A rubric is a floor: a Finding outside it is + allowed and counted, and is how the rubric grows. +- **Convergence also needs evidence, not only the absence of objection.** Where a work item has an + acceptance contract, every criterion in it must have passing evidence on the current content. + The contract is written before the work by a Role other than the one that implements it, is + approved by the maintainer, and cannot be changed by the implementing Role. It takes effect with + the fix and implement intents (design document, §3.13 and §4.8). +- **The required Lenses are declared per class of work item and stage, as data.** The starting set + is the deterministic checks, the in-house reviewer and humans. A Finding from a Lens outside the + required set gates only when a required Lens adopts it as its own. Model Lenses are admitted only + after the deterministic required Lenses have passed on that content. +- **A Lens earns its place.** The Ledger records each Lens's Findings, the share acted on, the share + no other Lens raised, and its cost. A Lens joins a required set only with a measured marginal + contribution. + +This decision fixes the contract and the measure. It does not choose the council. + +### Follow-on ADRs this decision requires + +Numbers are assigned when each is written. Each lands with the migration step that makes it true. + +| Follow-on | Must decide | Effect on the log | +|---|---|---| +| **Agent identity** | Each Role's App and its permissions (the shared name prefix is `combsmith-sf-`); how tokens are narrowed per Role; what the relay holds; the reviewer credential's confinement; what interaction model §12.2 says afterwards. | Supersedes ADR-0008. | +| **Records and the control room** | Run naming; the artifact and view schemas; where held-out eval sets live; how legacy markers are read during migration. | New. No ADR governs this today. | +| **Admission** | The poller and its schedule; the relay; the pass record; how a human Pause and a Hold are read; the LLM capacity signal and its credential (PRD OQ-3); priority order. | Replaces ADR-0007's file in each consumer with none: the poller reads every repo and the relay hears its events. Admission trusts no event: an Outside contributor's pull request produces no Run that holds secrets or writes without a Maintainer decision (PRD FR-63). ADR-0010's job-level concurrency key stays in use for the per-item group. Supersedes ADR-0009 when the router it names is retired. | +| **Run contract** | Task, Result and Effect schemas; the sandbox; resume; whether an adopted runtime fills the inner half. | Extends ADR-0005. Supersedes ADR-0011, which should be accepted now as the interim rule. | +| **Release unit and promotion** | The eval gate and its cost budget; ring membership; rollback; retirement of per-agent names. | Supersedes ADR-0002 and ADR-0006, restating what survives of each. | +| **Kernel language and verification** | Type checking; size and complexity budgets; recorded-interface tests; when the legacy suite becomes required. | Supersedes ADR-0004, restating its principle. | +| **Merge integrity** | The content identity; the gate function's inputs; which Lenses are required; the Override record; merge-queue settings. | New. No ADR governs this today. | +| **Factory repository** | The ruleset of `petry-projects/software-factory`; how the do-nothing commits in this repo are tested; the final re-pin of consumers; what this repo keeps afterwards; where these three documents and the ADR log live once the factory repo exists. | New. Decision 1 fixes that there is one. | +| **Council composition** | Which Lenses are required for each class of work item and stage (PRD OQ-12); how their Findings are aggregated; what counts as independent; the cost each stage may spend on its council. Decided from the Ledger's measure of each Lens, not in advance; review §16.6 proposes the set to start measuring from. | New. Extends ADR-0003, whose registry binds a review target to a rubric and is the nearest existing form of a Lens declaration. | + +ADR-0001 and ADR-0003 are kept without change. ADR-0012 governs the POC's comment gate until the +merge-integrity follow-on supersedes it. ADR-0000 is kept; adding a forward link for +extension remains a separate decision. + +## Consequences + +- **The failure classes get structural answers rather than guards.** A lost event delays a work + item instead of deleting its request. "Fixed" is written by the component that pushed the commit. + The Maintainer can review an agent's pull request. A change to any artifact reaches production by + one gated path. "Why is this not approved" has one function to read. + +- **It is a rewrite of the core, in a second language, while the line keeps running.** Roughly + 53,000 lines of shell and 70,000 lines of bats become legacy. Two systems coexist for months, and + the maintainer's load rises before it falls. The builders are largely the agents being replaced. + +- **Incident knowledge will be lost in translation.** The bats suite encodes years of specific + failures; one script cites 48 issues. Recorded-fixture replay recovers some of it. The comparison with + the POC's decisions in decision 10 is the only measure of how much. + +- **GitHub Actions stays the queue and the timer.** This decision bets that reconciling from + recorded state makes dropped ticks and evicted runs harmless. It does not make them rare. Every + pass costs a runner start, and a change with no nudge waits for the next pass. If the bet fails, + the off-Actions control plane ADR-0007 rejected is still to be built. The Kernel's lack of I/O is + what keeps that move cheap. + +- **Hub execution concentrates the blast radius.** A bad `stable` Release, or a broken entry shim, + stops every repo at once. ADR-0007 accepted that per repo; this accepts it per org. The shim's + byte freeze and the promotion gate are correspondingly load-bearing. + +- **Four repos for a while.** The standards repo, this repo, the factory repo and the control room coexist until the + last consumer is re-pinned. The split this review criticised gets wider before it gets narrower. + +- **Nothing limits the build, or the period in which two systems run.** Stopping is a judgement + the maintainer makes, the POC has no cap on fixes, and until a Release on `next` serves the factory repo + every factory change needs the Maintainer's approval. The review ranks a starved migration as the + likeliest failure (§14.2), and the period in which that would happen is the one these leave open. + The signal to watch is the weekly share of merged pull requests that land in the factory repo. + +- **Domain owners are model Lenses that gate.** Each owning Role's decision to accept a Disposition + closes a Finding. The research is plain that a reviewer weaker than the author makes the result + worse (review §15). Each owner therefore needs an eval set before it gates and is measured like + any other Lens. + +- **One App per Role means up to about ten private keys in the end,** each a high-value secret, + and a manual registration when a Role first writes. + +- **A greenfield build can fail to reach parity.** The POC's tests encode incident history that a + new codebase does not have. The comparison in decision 10 is the only measure of how + much was lost, and it measures decisions, not every effect. + +- **Cutover leaves a gap.** An intent family is unserved on a repo from the moment the POC stops + serving it until the factory's next Release, which the PRD bounds at 48 hours. + +- **The factory's own changes are gated by a Release that has not been promoted.** The factory repo + is served by `next`, because its own pull requests are the best evidence a Release can have. A + bad Release reaches the factory repo at once, and only that repo. The deterministic required + check, the brakes and the Maintainer's ability to block are what stand behind it. + +- **A consumer's work runs in the hub.** Runs for a consumer's work items execute and log in the + factory repo; the consumer's own Actions tab shows nothing of them, or only a nudge. `verify` runs the consumer's + tests on a hub runner, so each consumer must declare its setup and test commands, which the POC + infers. + +- **One Release removes per-role staging and rollback.** A prompt fix for one Role waits for the + whole gate, and a rollback reverts every Role. ADR-0006 made this trade for personas knowingly; + this makes it for the fleet. Two channels are also less staged exposure than the four ADR-0002 + allows. + +- **Eval-gated promotion spends the scarcest resource.** Each eval case is a real model call against + the subscription window. Candidates must be batched and results cached, so "merged" and "live" + stay hours apart by design. + +- **The factory can trigger itself.** Events authored by an App start workflows; events authored by + the default workflow token mostly do not, which ADR-0007 called a free brake. The POC's main + writer never had that brake, and after this decision nothing does. The brakes outside the + Kernel (decision 3) replace it. They are small, and they are all that stands between a Kernel + defect and an org-wide loop that could spend a week's model budget in hours. + +- **This decision does not raise the factory's ceiling.** Output today is bounded by how many + changes the maintainer can audit and merge by hand, and beyond that by how often a change is + right the first time. A control plane makes approvals issue reliably. It does not make them + correct. The PRD's autonomy and defect targets cannot both be met on current first-pass + correctness (review §14.4). An acceptance contract (decision 12) is the first answer to that. Review + depth is the ceiling after it and is not decided here. + +- **The largest failure class is mostly policy, not structure.** Approvals fail to issue chiefly + because every comment from nine third-party reviewers must be dispositioned. Decision 7 narrows + what gates an approval. Without that change the control plane would reproduce the same deadlocks + in one place instead of five. That change can also be made without this decision, in weeks, and + doing so first is the cheapest test of how much of the rest is needed (review §14.5). + +- **The credential boundary ends at the hub.** A branch pushed by `apply` runs the consumer's own + `pull_request` workflows with that repo's secrets. Decision 5 keeps factory secrets out of that + path. It does not protect a consumer's own secrets from a model-authored change, which is why + `apply` refuses workflow changes without a human transition. + +- **A poller is slower than an event.** A scheduled tick cannot be finer than five minutes and is + often dropped here, so a change with no nudge waits for the next pass. In exchange a flood of + events costs at most back-to-back passes, one runner slot of twenty. Decision 11 measures the + delay. + +- **Each App's private key is a high-value secret,** and the reviewer remains a user account with a + long-lived token. The second is a standing exception to least privilege, confined to one job. + +- **Re-issuing an approval for unchanged content approves a commit no reviewer read.** A defect in + the content identity is an unreviewed merge. It fails closed on any doubt, and it must be watched. + +- **The factory's history is 90 days.** That is how long GitHub keeps run records and artifacts in + a public repo. Nothing older can be replayed or audited. + +- **Money does not gate.** No decision reads the control room, and what a Run cost is known only + there. The ceiling that places a Hold is a number of Runs per work item; the Hold states the + amount spent. On a metered credential the monthly cap of decision 3 is read from the provider. + +- **A killed Run's cost is not known exactly.** The model CLI reports usage when an invocation + ends, so a Run that dies mid-turn has only its duration. + +- **The control room is public.** It aggregates on one page what is already public item by item, + including what awaits the maintainer and how much capacity is left. + +- **An injected model can publish the model credential without any network access.** The `run` job + has no network containment, and in a public repo its log and artifacts are world-readable. + Three measures reduce this and none removes it: no shell or network tool for a Run that reads + untrusted text, the model CLI's own sandbox if a trial shows it holds, and a credential that is + the factory's alone. An adopted runtime with a firewall would close more of it. + +- **A council costs what it finds.** Every model Lens spends from the same scarce window, and the + research is plain that adding one can lower quality: a weaker reviewer made a stronger author's + work worse in a controlled study, and agents that cooperate on one piece of code succeed less + often than agents working alone (review §15). Decision 12 therefore measures each Lens and keeps + one author. It does not assume that more Lenses are better. + +- **Some current reviewers lose their say.** A third-party reviewer that offers only a summary + comment has no contract surface. Under decision 12 it becomes presentation for the maintainer and + input a Role may read, and it gates nothing. + +- **Nine more ADRs.** Five of them supersede accepted records. That ceremony is the price of a log + that says what the system does. + +- **Not addressed:** review depth, the reliability of third-party advisory reviewers beyond a + deadline, and an exact LLM capacity signal, which the platform does not offer. + +- **Reopen an off-Actions control plane** when a threshold in decision 11 fails; or, later, when + more than 5% of admitted Runs are lost or start later than the liveness limit for four weeks; or + the hub's run queue is the measured bottleneck on lead time; or API quota cannot be held under + the PRD's headroom target; or a timer finer than a pass is required. +- **Reopen the worker runtime, the capacity model and eval-gated promotion** when the model CLI's + headless default stops reading subscription credentials, which its documentation says will + happen. +- **Reopen build versus adopt** when GitHub ships a generally available orchestrator that covers + admission and identity. +- **Reopen runner budgeting** if the free plan's concurrency limit or the pricing of public-repo + minutes changes. Each reopening means a new ADR. + +## References + +- `docs/prd/agentic-software-factory.md` — the requirements this decision is measured against. +- `docs/architecture/software-factory-design.md` — the interaction and contract design: structure, + contracts with examples, the Lifecycle and its invariants, Role authority, why State is derived + and not stored, how work is noticed, and the control room. +- `docs/architecture/factory-mvp-architecture-review.md` — evidence (§§2–3), root causes (§4), + options and sub-decisions (§§5–6), target and trade-offs (§7), migration shape (§8), relationship + to earlier ADRs (§9), requirements the analysis changed (§10), platform facts and assumptions + (§11), comparison with other systems (§13), the adversarial review and the ceiling (§14), the + evidence on councils and convergence (§15), reviewer surfaces and a starting council (§16), where + a Lens runs (§17), building in place against building greenfield (§18). +- ADR-0000 (process) — immutability and supersede-not-edit, which the precedence rule above follows. +- ADR-0001 (thin caller / reusable) — the boundary the entry shim keeps. +- ADR-0003 (review artifact contract and rubric registry) — the existing data contract that + decision 12 generalizes from review targets to Lenses. +- ADR-0002 (channel tags and rings) — the mechanism decision 6 reuses with one name. +- ADR-0004 (pure logic plus bats) — the principle decision 1 keeps and the binding it will replace. +- ADR-0005 (advisory by default) — the opt-in decision 4 enforces. +- ADR-0006 (shared-runtime personas) — "a persona is a caller of the shared runtime", generalized + to every Role. +- ADR-0007 (one ingress per repo) — the file this keeps, the hosted receiver it rejected, and the + free loop brake this gives up knowingly. +- ADR-0008 (personas post as the owner) and `docs/agentic-interaction-model.md` §12.2 — the + decisions the identity follow-on will supersede and reverse. +- ADR-0010 (ingress concurrency; allowlist schema) — the missing `extended-by` status. +- ADR-0011 (write-mode personas on a shared branch, proposed) — the interim rules decision 4 makes + structural. +- `docs/postmortems/2026-06-pr-860-runaway.md` — why a loop bound must not depend on a reset the + loop can reach. +- `scripts/lib/carry-forward.sh`, `.github/workflows/issue-triage-runner.yml`, + `.github/workflows/persona-runner-reusable.yml`, `scripts/initiative-planner/apply-plan.sh` — the + existing seams decisions 4 and 7 generalize. +- `petry-projects/.github-private#2197 — The fleet and every session share one GitHub API quota + (don-petry, 5,000/hour) and the fleet alone uses about that much: it ran out on 2026-10-10 and + failed a workflow`. +- `petry-projects/.github-private#1864 — Merge protections are unsatisfiable at N>1 open PRs: one + merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero + merges`. +- `petry-projects/.github-private#1741 — [bug] 42% of dev-lead runs are cancelled before doing any + work — review findings sit unaddressed while the PR looks active`. diff --git a/docs/architecture/factory-mvp-architecture-review.md b/docs/architecture/factory-mvp-architecture-review.md new file mode 100644 index 00000000..80428984 --- /dev/null +++ b/docs/architecture/factory-mvp-architecture-review.md @@ -0,0 +1,2363 @@ +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# Architecture review: the agentic factory, POC to MVP + +**Status:** Draft — for maintainer review. **This is a historical analysis, not a specification.** +Several of its recommendations were replaced on the day it was written. Build from +[`software-factory-design.md`](./software-factory-design.md) and ADR-0013, never from this. +**Date:** 2026-10-10 +**Reviewed at:** commit `aef61164` (`main`) +**Lens:** Solution Architect persona (`personas/solution-architect/`), run locally for a whole-system review +**Companions:** [PRD](../prd/agentic-software-factory.md) · +[ADR-0013 (proposed)](./adr/0013-factory-mvp-reference-architecture.md) + +## 1. Summary + +**The POC proved every function of the factory and has stopped converging.** In the last measured +week the hosting repo ran at least 44,968 workflows to merge 30 pull requests. Merges per week have +been flat since August while median time to merge rose from about one hour to between twelve and +twenty-two. No open pull request in the org holds an approval, and a third of recent merges had no standing +approval when they merged. Bug-labelled issues have risen from under ten a week to between twenty and thirty-six. +Eight fixes moved a failure somewhere else instead of removing it (§2.4). + +**Seven structural causes explain the eleven recurring failure classes** (§4). Each is a decision the +POC never made once, so it was made many times, locally: + +1. Work-item state is implicit: labels and about eighty kinds of comment marker, with no declared + lifecycle. +2. Work is carried by events and concurrency slots on a platform that drops both, and nothing admits + runs in one place. +3. The model holds the write credential and reports on its own work. +4. Agents act as the maintainer: one account, one credential, one API quota. +5. There is no single unit of release, and one fact is written in up to eight places across two + repos. +6. An approval is bound to a commit hash, can be withdrawn by five independent gates, and one merge + touches every open pull request. +7. Verification does not bind: the test suite, the validators and the evals block nothing. + +**Recommendation: restructure on GitHub Actions around a declared control plane, one run contract +and one release** (§7), recorded as ADR-0013. A level-triggered Controller reconciles each work item +from recorded state, so a lost event delays work instead of deleting it. Every role in scope for the MVP +runs under one contract in which the model requests effects and a separate job applies them. Agents write through +a GitHub App and the owner account becomes the human's alone. Everything ships as one release on one +gated path. No always-on infrastructure is added, and the decision logic is written so that it could +later run behind a hosted control plane unchanged. + +**The strongest argument against it** is that it keeps GitHub Actions as the queue and the timer, +which is the substrate behind the second-largest failure class, and bets that reconciling from +recorded state makes loss harmless. It is also a rewrite of the core in a second language, built +largely by the agents it replaces, on a line that is already overloaded (§7.5). + +**What should be kept** is listed in §3.7. The release-tag mechanism, self-hosting safety, personas +as data, the read/write split in the persona runner, the push guards and the ADR practice all carry +over. + +**The first two steps deliver value alone** (§8). Step 0 is configuration only: stop updating +branches that do not conflict, try the deeper concurrency queue GitHub now offers, pin the model +CLI, and make verification a required check. Step 1 runs the new decision logic read-only on every +repo, puts one status block on every work item, and produces the measurements the PRD's targets +need. + +**An adversarial pass then attacked the recommendation** (§14), and a comparison with seven other +systems tested it from outside (§13). Three findings changed the proposal: + +- **The architecture does not raise the ceiling.** Output is bounded first by how many changes the + maintainer can audit and merge by hand, about 30–36 a week in the hosting repo, and then by how + often a change is right the first time. The proposal removes waste under the machine limits. The + PRD's 85% autonomy target and its defect target cannot both be met on current correctness. +- **The largest failure class is mostly policy.** Approvals fail to issue chiefly because every + comment from nine third-party reviewers needs a verified disposition. Narrowing what gates an + approval would likely do more for that class than the rewrite, and can be tried in weeks. +- **Two mechanisms were a bet.** Every comparable orchestrator polls from one standing process and + keeps state it can rebuild. Files in git as a work ledger is a design another project abandoned. + Both are now provisional in the ADR, with thresholds, and step 1 runs the same logic under a + poller for comparison. + +The same pass found that the migration is the likeliest thing to fail: holding the POC still needs +more fix pull requests a week than the line delivers. The ADR therefore required a dated limit, a +defect policy and a kill criterion before it was in force. The maintainer set them on 2026-10-10 +(§18.5). + +**The maintainer's core hypothesis is convergence**: that a council of models, tools and +perspectives produces a better outcome than one agent alone. The first draft did not evaluate it. +§15 does. The evidence supports it under conditions: an independent evaluator beats +self-evaluation, and Lenses that can see different things find different defects; but a council +that converses does worse than one that reports, a council that co-writes does worse than one +author, and a weaker reviewer makes a stronger author's work worse. The POC has run a large council +for months through free-text discussion and cannot say what any member contributed. ADR-0013 now +gives every Lens, third-party reviewers included, one structured contract, decides convergence in +the Kernel, and measures each Lens before any is added. + +**Decisions this asks of the maintainer** are in the PRD, §11. The three that shape everything else: +whether agents move off the owner account, whether the POC is frozen for features while it is +replaced, and what the migration may cost. A fourth was decided on 2026-10-10: third-party +reviewers are integrated through a contract, and comment bodies gate nothing. The same day the +maintainer decided that no agent runs as the owner account, named the new repo +`petry-projects/software-factory`, settled the migration's governance, and chose which +reviewers stay (§16.6), and chose one App per Role. The interaction and contract design that +follows from these is in `software-factory-design.md`. It replaces two recommendations made here: +the git-backed store of §6.1 gives way to derived state, Actions run records and a rebuildable +view, and the wake per event of §6.2 gives way to a poller. + +The rest of this summary is the review as first written. Where it says the two mechanisms are +provisional, that step 1 runs read-only on every repo, or that the migration has a limit and a kill +criterion, ADR-0013 and the design document are current: both mechanisms are settled, the first +component is the review intent on the factory's own repo, and there is no limit and no kill +criterion. + +**It will be built in a new repo and cut over through the existing channel tags** (§18). Building in +place would put every factory change through the POC's own line, which is the thing that is +failing. A literal swap is not available, because GitHub does not redirect reusable-workflow +references. The hybrid builds greenfield, runs read-only against real traffic from its first weeks, +and switches one intent and one ring at a time by moving a tag. + +**Which reviewers have a contract** is answered in §16. Of the nine installed, SonarQube Cloud has a +full one, and the POC reads its comment instead; cubic has a findings API; CodeRabbit has a +structured verdict; the rest offer review threads or nothing. No LLM reviewer's severity is +readable without parsing a comment body. §16.6 proposes a council to start measuring from: six +deterministic and execution-grounded Lenses on one contract, the in-house reviewer, and three or +four of the nine outside reviewers. + +Fifteen defects found in passing are listed in §12. They are worth fixing whatever is decided. + +## 2. Evidence base + +### 2.1 Sources and method + +Four read-only passes were made over the repo and its telemetry, each producing claims tagged +*verified* (the code or data was read) or *inferred*: + +1. **Execution harness** — everything between "a job decided an agent should run" and "its output + was applied": `scripts/engine.sh`, `scripts/lib/`, the dev-lead, pr-review and persona runners. +2. **Interaction model and control plane** — `docs/agentic-interaction-model.md`, the interaction + contracts, the ingress and repair workflows, the gates. +3. **Distribution and assurance** — release tooling, the standards repo `petry-projects/.github`, + evals, tests, validators. +4. **Telemetry and issue history** — in-repo reports, 600 issues and 400 pull requests from the + hosting repo, 300 issues from the standards repo, seven days of workflow runs, the org-wide open + pull-request backlog. + +The ADR corpus (ADR-0000 to ADR-0011), the interaction-model standard and the operating rules in +`AGENTS.md` were read in full. The highest-consequence claims were re-checked against the code and, +for the branch ruleset, against the live API. + +Citations are `path:line` at the reviewed commit. A `DG:` prefix marks a path in the standards repo, +`petry-projects/.github`, at commit `3cd1511f`. Issues are cited as `repo#number — title`. + +### 2.2 Limits + +- **Shared identity hides the human.** dev-lead, the advisory personas, interactive sessions and the + maintainer all act as `don-petry`. Every non-Dependabot pull request in the sample is authored and + merged by that account. "Agent-authored" is inferred from the branch prefix. No measurement of + human intervention is exact; the figures given are bounds. +- **Run counts are floors.** The runs API caps a count at 2,500 per query. Per-workflow rates come + from a 2,730-run sample that under-represents bursts. +- **Issue classification is by title pattern.** Classes overlap. Counts show relative frequency. +- **Consumer repos were not inspected live.** Local checkouts are twelve weeks old. Statements about + what consumers pin come from in-repo documents dated 2026-09-29 or later. +- **Channel-tag facts are from local tags** fetched 2026-10-10, not from the remote at query time. +- **Not measured:** runner minutes; cost per merged change; review quality (the scorecard's miss + rate reads 0 of 0); the cause of 410 failed runs. + +### 2.3 Failure classes + +Classification of 203 defect and process-gap issues in the hosting repo and 117 in the standards +repo. "Last 4 / prior 4 / 4 before" are four-week windows ending 2026-10-11. + +| Class | Hosting repo (open) | Last 4 / prior 4 / 4 before | Standards repo (open) | How it has been handled | +|---|---:|---|---:|---| +| A. Approval never issues, or issues and is withdrawn | 63 (20) | 45 / 17 / 1 | 6 (2) | Patched gate by gate; the next gate then failed | +| D. Work dropped between event and agent | 47 (19) | 31 / 16 / 0 | 17 (5) | Sweeps and retries, which suffer the same loss | +| B. Status reported but not real | 37 (15) | 18 / 16 / 3 | 14 (8) | Verifiers on agent output, which produced their own deadlocks | +| E. Quota and budget exhaustion | 33 (17) | 24 / 8 / 1 | 6 (3) | Control built, closed as done, cannot fire | +| C. Merged fix is not what runs | 25 (12) | 13 / 9 / 3 | 72 (33) | Manual override promotions | +| F. Identity, credential, permission | 21 (8) | 9 / 6 / 4 | 3 (0) | Relocated three times | +| G. Re-review churn, rebase treadmill, runaway | 17 (5) | 11 / 6 / 0 | 4 (2) | Per-pull-request caps; cross-pull-request cost unbounded | +| I. Human steering does not reach the agent | 14 (5) | 8 / 6 / 0 | 12 (4) | Partly fixed; the working recipe is procedural | +| J. Tests and evals not measuring the real thing | 14 (7) | 7 / 5 / 2 | 16 (6) | Case by case | +| K. Automation that corrupts or destroys | 13 (6) | 7 / 5 / 1 | 8 (4) | One required-check guard; variants recur outside its scope | +| H. Timeout loses the work | 9 (7) | 2 / 7 / 0 | 3 (3) | Checkpoint push exists; resume does not | + +Every class except H grew in the most recent four weeks. Representative issues per class: + +- **A.** `petry-projects/.github-private#1894 — 10 of 11 open PRs have never received an approving + review`; `#1933 — pr-review never re-approves after dismissing its own approval`; `#2166 — + Edited-after-disposition deadlock`. +- **D.** `petry-projects/.github-private#1741 — [bug] 42% of dev-lead runs are cancelled before + doing any work`; `#2017 — A lost fix-bot-comment run is never retried`; `#2170 — Retry and sweep + crons run every 4–8 hours instead of every 2 h / 15 min`. +- **B.** `petry-projects/.github-private#1620 — dev-lead merges zero-diff PRs that close their + issues`; `#1874 — pr-review announces an approval it never created`; `#2013 — dev-lead + fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests to match + its own code`. +- **E.** `petry-projects/.github-private#1565 — Auto-pause companion: usage-telemetry adapter + + scheduled poller`; `#1862 — check_provider_headroom has never worked`; `#2197 — The fleet and + every session share one GitHub API quota`. +- **C.** `petry-projects/.github-private#1592 — Script-only changes never reach the fleet`; + `petry-projects/.github#1125 — Fleet pr-review callers pin legacy bare pr-review/stable and are + unmanaged`; `petry-projects/.github#1086 — Gate soak is vacuous`. +- **F.** `petry-projects/.github-private#1415 — [Phase 4] Close the maintainer review-gate gap`; + `#1734 — persona-runner: qa-lead advisory lost`; `#2027 — dev-lead silently ignores @dev-lead + mentions from the maintainer, because the maintainer's account is dev-lead's own identity`. +- **G.** `petry-projects/.github-private#1864 — Merge protections are unsatisfiable at N>1 open + PRs`; `#1881 — auto-rebase still merges main into every open PR on every push`. + +### 2.4 Fixes that relocated the failure + +| Fix | What it resolved | Where the failure moved | +|---|---|---| +| Channel tags and rings (June–July) | Unversioned consumers; an agent could break its own merge gate | Merged fixes do not run; rings with no members; manual overrides | +| Restrict auto-rebase to review-ready pull requests (June) | Fan-out from 170–220 runs a day to about 6 | Reversed in August to fix approval currency; fan-out back by September | +| SHA-keyed concurrency (August) | 49% pr-review cancel rate | 37% on pr-review and 42% on dev-lead | +| Maintainer-comment disposition gate (September) | Unanswered maintainer comments merging | Gate blocks on bot notices, the agent's own acknowledgements, edited summaries | +| Claim verification guards (September–October) | False "applied" and phantom "Fixed" | Guards refuse legitimate pushes, loop, or retract real fixes | +| Persona posting identity (September) | Advisories under the owner credential | HTTP 403 on the bot credential; reverted to the owner account | +| Drop the strict up-to-date rule (2026-09-21) | Pull requests stuck behind `main` | Auto-rebase kept merging `main` into every open pull request (`petry-projects/.github-private#1881 — auto-rebase still merges main into every open PR on every push`); the compliance audit now reports the change as drift (`petry-projects/.github#1265 — Compliance audit — 2026-10-09`) | +| Budget breaker epic (closed 2026-09-07) | Nothing | Breaker unreachable; poller credential refused | + +Genuine reductions: hard failures are 0.9% of runs against 100% failure streaks in April +(`docs/pr-review-agent/failure-report.md`); the self-mention loop behind the PR 860 runaway has not +recurred (`docs/postmortems/2026-06-pr-860-runaway.md`); no trunk corruption since +`duplicate-decl-gate` became a required check. + +## 3. Current-state findings + +### 3.1 What the system is + +A set of LLM agent roles hosted on GitHub Actions and delivered to consumer repos through caller +stubs that pin reusable workflows by channel tag. + +| Fact | Value | +|---|---| +| Repos served | 12, all public | +| Shell in `scripts/` | 182 files, about 53,400 lines, 1,107 functions | +| Workflow YAML | 62 files, about 10,200 lines, of which about 1,960 are inline shell | +| Tests | 229 bats files, about 69,600 lines, about 4,940 cases | +| Scheduled workflows | 22, with 24 cron expressions | +| Roles | 9 persona manifests, all `status: draft`, including the two that run fleet-wide | +| Secrets referenced | 18 names; `GH_PAT_DON_PETRY` 70 times | +| Third-party review bots tracked | 9 | + +The system spans two repos. The hosting repo holds the runtimes, scripts, prompts, personas and +evals. The standards repo holds the mention routers, the ring registry and the release manager that +govern them (`AGENTS.md:955-961`). A persona behaviour change can need a pull request in each: the +router in the standards repo reads persona manifests from the hosting repo's `main` and dispatches +back to it. + +### 3.2 Control plane: no declared lifecycle, and the machinery that compensates + +**No file states the lifecycle of a work item.** What is declared is narrower: trigger classes per +workflow (`docs/agentic-interaction-model.md:185-224`), dev-lead intents +(`scripts/dev-lead-intent.sh:9-19`), and pr-review verdict reasons as free strings +(`scripts/review-one-pr.sh:333` and eleven further sites). The actual state of a pull request is a +tuple of labels, HTML-comment markers, review objects, review decision, check rollup, head commit, +minimized-comment state and merge-queue membership, recomputed from the API by each script on each +run. Transitions are spread across workflow `if:` expressions, the intent classifier and about +twelve sequential gates in `review-one-pr.sh`. + +Consequences that follow directly: + +- **Markers are the state store, with no schema.** About 58 to 80 distinct marker prefixes exist. + The only registry is one prefix regex (`scripts/lib/maintainer-comment-gate.sh:74`). A marker can + outlive the object it describes, which is why `scripts/lib/standing-approval.sh:2-15` exists. +- **Approval is a conjunction, not a state.** It is a non-dismissed review by the bot at the head + commit *and* a marker. Five gates can each withhold or dismiss it. +- **Releasing a hold is not an event.** dev-lead's stub has no `unlabeled` trigger and pr-review + subscribes to no label events (`.github/workflows/pr-review-trigger.yml:27-53`). A released hold + takes effect on the next unrelated event. +- **One label has four meanings.** dev-lead stops on `needs-human-review` unconditionally + (`scripts/lib/hold-gate.sh:32`). The pr-review cascade stops only if an escalation marker is also + present (`scripts/review-one-pr.sh:927-941`). The sweep stops unless the hold is rate-limit-only. + The initiative driver never releases. `AGENTS.md:71-88` documents this as a rule for operators. +- **The pause switch pauses one role.** `AGENTS_PAUSED` appears in three `if:` lines, all in + `dev-lead-reusable.yml` (`:116`, `:642`, `:748`). `pr-review.yml:344` tells operators to use it + and does not read it. + +**Concurrency groups are the work queue, and the queue holds one item.** GitHub keeps one running +and one pending run per group; a third arrival evicts the pending one. Both main agents document +this (`.github/workflows/dev-lead-reusable.yml:76-85`; `.github/workflows/pr-review.yml:138-144`). +Fan-in makes it routine: one `issue_comment` starts three agentic workflows; one submitted review +starts four. A dev-lead run killed before posting its marker is never retried, because the retry +scan reads the absence of a failure marker as "nothing failed" +(`scripts/dev-lead-retry.sh:1307-1311`). + +**Fifteen workflows exist to retry, sweep, repair, audit or detect**, against twenty-eight that do +primary work. In the dev-lead and pr-review loop the ratio is twelve to seven. Each stands in for an +invariant the primary flow does not hold: + +| Compensating workflow | Missing invariant | +|---|---| +| `pr-review-sweep.yml` | A skipped review is not re-armed when the reason for the skip clears | +| `pr-review-delayed-retry.yml` | There is no durable timer; a runner sleeps until the rate-limit reset | +| `dev-lead-retry.yml` (four scans) | Work is not queued; a failed, rate-limited or evicted run is gone | +| `dismiss-stale-bot-reviews.yml` | A blocking review is not bound to the commit it reviewed | +| `repair-pr-approvals.yml` | Announcing an approval and creating it are not atomic | +| `auto-rebase-retry.yml` | A failed run leaves no signal for anything to retry | +| `auto-rebase-health.yml` | The trigger is an unindexed comment sentinel | +| `daily-pr-review-health.yml` (runaway, stall, merge-ready scans) | Loop bounds are not trusted; no liveness guarantee; approved does not imply merged | +| `premature-closure-audit.yml` | "Done" is asserted by a comment, not derived from a merge | +| `actions-fleet-monitor.yml` and the stub-drift tools | Control-plane configuration is distributed by copy | +| `engine-token-liveness.yml` | A run without a token can exit successfully having done nothing | +| `pr-review-canary.yml`, `initiative-driver-canary.yml`, `initiative-planner-canary.yml` | Merged code is inert until a tag moves; breakage is found on real work | +| `budget-poller.yml` | Capacity is not checked before a run starts | + +A further twenty-five or so scripts and libraries do the same inside primary workflows: +`carry-forward.sh`, `self-dismissed-approval.sh`, `verify-approval-review.sh`, +`merge-queue-dequeue.sh`, `claim-landing.sh`, `addressed-claim-verify.sh`, `rebase-exhaustion.sh` +and others. + +**The authoritative loop bound cannot see the main writer.** The per-pull-request automation budget +counts bot events since the last human event. A login is a bot if it ends in `[bot]` or is one of +`donpetry-bot`, `github-actions[bot]`, `repair-pr-approvals` +(`scripts/lib/pr-automation-budget.sh:32`). dev-lead commits and comments as `don-petry` +(`personas/dev-lead/persona.yml:107-109`). Its activity therefore classifies as human and resets the +count. No workflow overrides the list. The interaction model calls this budget authoritative and +states the rule it implements: a cap whose reset is reachable by the runaway is not a cap +(`docs/agentic-interaction-model.md:421-422`). The consequence is inferred from the code and was not +observed in a run. + +**The validators check event names, not behaviour.** They cannot see `if:` logic, the intent +classifier, payloads, or whether a stop marker is read. `pr-review:hands-off` is a declared stop +marker (`interaction-contracts/pr-review.yml:65`) that no code in this repo reads. Eighteen +declared-versus-deployed divergences were found, including: ADR-0007 is accepted and this repo has +no `agent-ingress.yml`, so `scripts/validate-ingress-if.sh:34-37` passes vacuously; the interaction +model's row for `initiative-planner.yml` says it cannot subscribe to `discussion: labeled` while the +workflow does (`.github/workflows/initiative-planner.yml:61-62`); dev-lead's `pull_request` surface +is declared advisory (`personas/dev-lead/persona.yml:64-68`) while those events route to an intent +that can commit and push (`scripts/dev-lead-intent.sh:432`, `:450`). + +### 3.3 Execution harness: five runtimes and a credential in the model + +**There are five ways an LLM is invoked.** `scripts/engine.sh` serves dev-lead, pr-review, +spec-drift and the evals. Personas call `claude --print` directly from workflow YAML +(`.github/workflows/persona-runner-reusable.yml:374`), as does the CI failure analyst +(`.github/workflows/ci-failure-analyst-reusable.yml:162`). Eight scripts call it directly from bash +(for example `scripts/aw.sh:365`). The initiative planner and idea triage use +`anthropics/claude-code-action` (`.github/workflows/initiative-planner.yml:246`). The pinned `gh-aw` +is used only to compile-check; `scripts/aw.sh` is a 586-line local reimplementation that serves one +workflow. + +Duplication follows from that: the cross-provider fallback loop exists twice +(`scripts/engine.sh:2082-2267`; `scripts/review-batch.sh:379-404`); the rate-limit pattern has five +independent definitions; the Claude Code install block appears fourteen times; prompts are assembled +by four different mechanisms. One of them is demonstrably wrong: the analyst's template passes +through unscoped `envsubst`, which blanks the shell variable the prompt tells the model to use, so +the model is told to run `gh run view ""` +(`.github/workflows/ci-failure-analyst-reusable.yml:115-117`, `:159`). + +**In the two busiest roles the model holds a write credential.** dev-lead's job sets the owner +credential for the whole job (`.github/workflows/dev-lead-reusable.yml:173`) and the checkout +persists it for `git push`. pr-review's tool-using tiers run with a classic `repo`-scoped token +(`.github/workflows/pr-review.yml:174-178`). "Never push" and "never resolve a thread" are prompt +text (`prompts/dev-lead/fix-reviews.md:58`, `:115-117`). The repo's own comments say what that is +worth: "a boundary that lives solely in the model prompt is not a boundary" +(`scripts/engine.sh:1843-1845`). + +The cost of that choice is visible in the code. The function that invokes the model in +`scripts/dev-lead-fix-reviews.sh` is 54 lines (`:152-205`) of a 3,989-line script. Most of the rest +verifies, attributes and retracts what the model did with its own write access: +`resolve_dispositioned_comments` alone is 473 lines (`:1533`). + +The persona runner is the exception and the model for the rest. The model gets a read-only token and +prints between sentinels; the wrapper is the only writer, enforces the marker and keeps the paid +output as an artifact if posting fails (`.github/workflows/persona-runner-reusable.yml:14-21`, +`:128-140`). Issue triage does the same with a separate job holding the write permission. + +**Resource governance is mostly inert.** The proactive Claude headroom probe needs +`ANTHROPIC_API_KEY`, which no workflow supplies, so it logs "usage 0% — ok" on every call +(`scripts/engine.sh:500-517`, `:591`). Every Gemini cap ships as `unknown` +(`scripts/lib/gemini-quota-caps.tsv`). The budget poller is dry-run by construction +(`.github/workflows/budget-poller.yml:8-10`) and its credential is refused. Nothing checks capacity +at the point a run is admitted. Token metering records only calls that succeeded +(`scripts/engine.sh:1801`, `:1962`), in three workflows, and attributes the wrong model after an +in-chain fallback (`:1740-1745`). + +**Complexity.** Seventeen functions exceed 150 lines. `scripts/review-one-pr.sh` has 1,025 lines of +top-level code that cannot be called as a unit. Control flow rides on overloaded exit codes (2, 3, +4, 55, 100, 101, 124). GNU-only constructs are on the hot path (`date -d` at 49 sites), and the +suite cannot run on stock macOS (`AGENTS.md:174-181`). The design rationale lives in code comments: +`scripts/dev-lead-fix-reviews.sh` cites 48 distinct issue numbers. + +### 3.4 Identity: one account, three kinds of actor + +The owner account is the maintainer, dev-lead, and all eight advisory personas (ADR-0008). pr-review +is the machine user `donpetry-bot`. No agent uses a GitHub App installation token; the two Apps in +use serve Dependabot rebasing and project boards. + +What this costs, each verified in code or in an issue: + +1. The maintainer cannot review dev-lead's pull requests. GitHub rejects an approval or a change + request on one's own pull request (`docs/agentic-interaction-model.md:719`). +2. The router cannot exclude agent output by actor, so an HTML marker is the sole mechanical + recursion guard for personas (ADR-0008, Consequences). +3. dev-lead cannot tell its own replies from the maintainer's by login and needs a `PATH`-shimmed + `gh` to attribute them (`scripts/lib/claim-landing.sh:115-170`). +4. The automation budget cannot count dev-lead (§3.2). +5. dev-lead ignores the maintainer's mentions as self-comments (`petry-projects/.github-private#2027 + — dev-lead silently ignores @dev-lead mentions from the maintainer`). +6. The fleet and the maintainer's own sessions draw on one API limit, which ran out on 2026-10-10 + (`petry-projects/.github-private#2197 — The fleet and every session share one GitHub API quota`). +7. No measurement can separate what a human did from what an agent did (§2.2). + +The interaction model, §12.2, weighed giving dev-lead a distinct identity and rejected it in August +on four grounds: code-owner and ruleset changes, a wider credential surface, churn in identity +resolution, and that a marker would still be needed for other automation posting as the owner. Items +2, 4, 5, 6 and 7 above were found after that analysis. + +### 3.5 Release, distribution and assurance + +**Two opposite release regimes coexist.** A change to dev-lead or pr-review is inert until a channel +tag moves (`AGENTS.md:25-33`). A change to a persona prompt or manifest is live for every repo that +reaches the router the moment it merges, with no ring, soak or gate: +`.github/workflows/persona-runner.yml:39` calls the runtime by local path and the checkout takes no +ref. `AGENTS.md:41-43` tells authors that a change under `prompts/` or `personas/` rides the next +autocut, which is not true for the shared persona runtime. + +**The version name is per agent; the version is the whole repo.** A release tag points at a +whole-repo commit (`docs/release/versioning.md:26-28`), and autocut cuts an agent whenever any file +under `scripts/`, `prompts/` or `personas/` changes. In thirty days, 116 commits to `main` produced +46 dev-lead, 32 pr-review and 40 ci-failure-analyst releases. ADR-0006 drew the conclusion for +personas: per-persona tags "never delivered isolated rollback either — they only appeared to." The +same holds for per-agent tags. dev-lead majors 2 to 138 have exactly one release each, so "pin once" +does not survive a major. Workflow and scripts version together only because each stub writes the +tag twice, as `uses:` and as `agent_ref:`; `agent_ref` defaults to `main` when omitted +(`.github/workflows/dev-lead-reusable.yml:11-17`). + +**Promotion reads the signal the operating rules call unreliable.** The gate samples workflow run +conclusions. `AGENTS.md:48-55` says agents report success for work they did not do and instructs +operators to confirm the artifact. At the snapshot all four dev-lead channels pointed at one commit +and all four pr-review channels at another, and the gate had been overridden by hand on four dates +in ten days. `ci-failure-analyst/v1-stable` is 322 commits behind `main` while this repo's own stub +pins it. + +**One fact is written in many places.** "Which agents exist" appears in at least eight places across +two repos, and they already disagree: `scripts/cut-release.sh:70-89` lacks three registered agents. +A persona's trigger surface is written in up to eight places; the commit that subscribed one stub to +one event changed eight files. Thirty-odd validators exist largely to keep such copies in agreement. +Giving qa-lead a `pull_request` surface took 32 days and at least six pull requests across two +repos, and the surface is still served twice (ADR-0009). + +**Nothing in the assurance chain blocks.** + +- The live ruleset on `main` requires five checks: SonarCloud, CodeQL, AgentShield, + dependency-audit and `duplicate-decl-gate`. `Lint`, which runs the bats suite and every validator, + is not among them. The interaction model describes itself as "CI-verified, not review-verified" + (`docs/agentic-interaction-model.md:574-577`); CI does verify it, and a failure blocks nothing. +- `scripts/evals/score-gate.sh` runs only under `continue-on-error` + (`.github/workflows/skill-eval-report.yml:204`). The comment says the hard bar is enforced at + release-cut time; `scripts/cut-release.sh` does not reference it. +- The eval sets for dev-lead and pr-review score their mention-advisory prompts + (`scripts/evals/run-eval.sh:126-147`). The flows that write code and approve pull requests have no + eval. +- Four bats files run in no workflow. The bats list in `lint.yml` is maintained by hand, and + `scripts/run-bats.sh:5-11` warns and skips a listed file that does not exist. +- Many tests stub `gh` with a shim that returns post-filter output or accepts any flags, so the + script's real flags and filters are not exercised. + +### 3.6 Decision coverage + +| ADR | Status | State in this repo | +|---|---|---| +| ADR-0001 thin caller and reusable | accepted | In force; several "stubs" are not thin | +| ADR-0002 channel tags and rings | accepted | In force for three agents; not for personas or hub workflows | +| ADR-0003 rubric registry | accepted | In force | +| ADR-0004 pure logic and bats | accepted | Followed at function level; not at script level (§3.3) | +| ADR-0005 advisory by default | accepted | Declared; the `gate_label` is not enforced by code | +| ADR-0006 personas not ring-registered | accepted | In force; its own validator contradiction is unresolved | +| ADR-0007 one ingress per repo | accepted | Not deployed here; one consumer pilot | +| ADR-0008 personas post as owner | accepted | In force | +| ADR-0009 qa-lead surface via router | accepted | Transitional; served twice | +| ADR-0010 ingress concurrency | accepted | Tooling for a file this repo does not have | +| ADR-0011 write-mode on a shared branch | proposed | Implemented ahead of acceptance | + +Most load-bearing behaviour has no ADR. No recorded decision governs: state kept in labels and +markers; concurrency groups as the work queue; the shared owner identity (decided in a document +section, not an ADR); the automation budget; the hold taxonomy; the pause switch and its scope; the +break-glass; the maintainer-comment gate; merge-queue adoption (the decision record says deferred +and the queue is on); approval carry-forward; the org cap on automation pull requests. + +### 3.7 What is solid + +These are worth keeping as they are. Each is specified, tested and has held up. + +- **Immutable release tags plus moving channel tags under a ruleset**, moved through one API path by + an App identity (ADR-0002; `scripts/cut-release.sh:43-52`). +- **Self-hosting safety**: this repo's merge gate runs a promoted channel, enforced by a test that + parses the pin (`docs/release/versioning.md:81-95`). +- **Persona as data on a shared runtime** (ADR-0006): a new advisor is about nine data files in one + pull request. +- **The read/write split in the persona runner** and the staged, allow-listed apply in issue triage. +- **The schema-validated plan with an LLM-free applier** in the initiative planner + (`scripts/initiative-planner/plan.schema.json`, `apply-plan.sh`). +- **Push safety**: `push_no_clobber` and the push-landed read-back + (`scripts/lib/git-push-guard.sh`, `scripts/lib/claim-landing.sh`). +- **Small post-hoc guards on a model's diff**: net-diff, test-tamper, test-regression. +- **Engine configuration as tables**: tier-to-model chains with a lint against pinned versions + (`scripts/lib/engine-models.sh`, `scripts/check-model-pins.sh`). +- **ADR-0004's pure-function discipline and the fail-loud rule**, with 1.46 lines of test per line + of script. +- **The ADR practice itself**: each record names a boundary a check can assert and is honest about + cost. +- **The seven rules from the PR 860 postmortem** (`docs/agentic-interaction-model.md` §7). + +--- + +## 4. Root causes and the decisions they force + +The eleven failure classes in §2.3 are not eleven problems. Seven structural causes explain them. +Each is a decision the POC never made, so it was made many times, locally, in different ways. Where +no ADR governs a cause, that is stated: this section does not treat preference as recorded doctrine. + +### 4.1 The seven causes + +| # | Root cause | Evidence | Governing ADR today | +|---|---|---|---| +| RC1 | **Work-item state is implicit.** No file declares the states or transitions. State is a tuple recomputed by each script from labels, about eighty marker shapes, review objects, checks and the head commit. | No lifecycle file (`docs/agentic-interaction-model.md:185-224` declares trigger classes only). Marker grammar is "the `printf` that writes it and the `grep` that reads it"; the only registry is one regex (`scripts/lib/maintainer-comment-gate.sh:74`). "Is this item held?" has seven implementations with different label sets (`scripts/lib/hold-gate.sh:22-34`, `scripts/review-one-pr.sh:928`, `scripts/lib/pr-automation-budget.sh:96-104`, `scripts/qa-lead-advisory-gate.sh:32,75`, `scripts/pr-stall-detect.sh:57`, `scripts/initiative-driver.sh:75-76`, and the cross-repo router). | None on record. | +| RC2 | **Work is carried by events and concurrency slots on a platform that drops both.** A request exists only as a queued run. Nothing admits runs in one place, so capacity is discovered by failing. | "A queued run superseded in its concurrency lane is dropped with zero steps" (`.github/workflows/dev-lead-reusable.yml:76-85`; `.github/workflows/pr-review.yml:138-144`). A killed issue run leaves no marker and the retry scan reads that as nothing to do (`scripts/dev-lead-retry.sh:1307-1311`). Cron fires every 3–7 h instead of 15 min (`.github/workflows/pr-review-sweep.yml:15-18`). No admission-time reader of the budget feed exists (`scripts/dev-lead-retry.sh:168-288` is the only consumer). Fifteen workflows compensate (§3.2). | None. The interaction model §§5–6 is normative but not an ADR. | +| RC3 | **The model holds the write credential and reports on its own work.** Five runtimes, four prompt-assembly mechanisms, seven return channels; in the two busiest roles the harness polices claims after the fact. | dev-lead: owner PAT in the job environment, tools `Bash,Read,Write,Edit,…` (`.github/workflows/dev-lead-reusable.yml:173`; `scripts/engine.sh:1915-1916`). pr-review: a classic PAT with `repo` visible to tool-using tiers (`.github/workflows/pr-review.yml:174-178,442`). The function that invokes the model is 54 lines of a 3,989-line script (`scripts/dev-lead-fix-reviews.sh:152-205`); claim verification alone is 627 lines (`scripts/lib/addressed-claim-verify.sh`). | ADR-0005 (write is opt-in) and ADR-0011 (proposed) govern behaviour around the write. Nothing governs where the credential sits. | +| RC4 | **Agents act as the Maintainer.** One account is the human, dev-lead and eight personas; one PAT serves seven purposes; one API quota serves the fleet and every session. | `personas/dev-lead/persona.yml:107-109`; ADR-0008. The loop ceiling classifies by login, so dev-lead's own activity counts as human and resets it (`scripts/lib/pr-automation-budget.sh:32,54-59`). Replies need a PATH-shimmed `gh` to be attributed (`scripts/lib/claim-landing.sh:116-120`). `petry-projects/.github-private#2197 — The fleet and every session share one GitHub API quota (don-petry, 5,000/hour) and the fleet alone uses about that much: it ran out on 2026-10-10 and failed a workflow`. | ADR-0008 (accepted) and interaction model §12.2 decided this deliberately. | +| RC5 | **There is no single unit of release, and one fact is written in many places across two repos.** dev-lead and pr-review changes are inert until a tag moves; persona prompts, manifests and hub workflows are live at merge. | `AGENTS.md:25-33` against `.github/workflows/persona-runner.yml:23-24,39`. The tag name is per agent but points at a whole-repo commit (`docs/release/versioning.md:26-28`). "Which agents exist" is written in at least eight places and a persona's trigger surface in up to eight (§3.5); subscribing one stub to one event changed eight files (commit `49770289`). Promotion reads run conclusions (`DG:scripts/canary-rollout.sh:31-35`). | ADR-0001, 0002, 0006, 0007. They conflict in effect: 0002 gates reusables, 0006 exempts personas. | +| RC6 | **An approval is bound to a commit hash and can be withheld or dismissed by five independent gates; one merge touches every open pull request.** | Twelve sequential gates (`scripts/review-one-pr.sh:333` through `:1024`). The reviewer dismisses its own approval and then skips as already reviewed (`scripts/lib/self-dismissed-approval.sh:2-13`). Each base merge mints a new head and forces a full re-review (`scripts/review-one-pr.sh:838-842`); auto-rebase updates every behind pull request by default (`.github/workflows/auto-rebase.yml:25-30`). The queue checks reviews only on entry (`scripts/lib/merge-queue-dequeue.sh:5-8`). | None on record. Merge-queue adoption is "deferred to a human" in its own decision document (`docs/initiatives/auto-rebase-vs-merge-queue.md:3-9,133`) while already in operation (`AGENTS.md:401-466`). | +| RC7 | **Verification does not bind, and the decision logic lives in a medium that resists it.** | The `main` ruleset requires five checks; `Lint`, the bats suite and every validator are not among them (live ruleset, read 2026-10-10). Evals never block (`.github/workflows/skill-eval-report.yml:176,204`) and score advisory prompts, not the code-writing paths (`scripts/evals/run-eval.sh:126-147`). The shared `gh` stub returns empty for unknown calls (`tests/dev-lead/fixtures/stubs/gh:28`). The suite fails on stock macOS (`AGENTS.md:174-181`). 1,025 top-level lines of `scripts/review-one-pr.sh` cannot be unit-called. | ADR-0004 prescribes pure bash functions plus bats. It does not say a test must block anything. | + +### 4.2 Failure classes to causes + +| Class (§2.3) | Primary cause | Contributing | PRD requirements | +|---|---|---|---| +| A. Approval never issues or is withdrawn | RC6 | RC1, RC4 | FR-2, FR-29 to FR-33 | +| D. Work dropped between event and agent | RC2 | RC1 | FR-3, FR-5, FR-15, FR-16, FR-19 | +| B. Status reported but not real | RC3 | RC1, RC7 | FR-4, FR-10, FR-11, FR-47 | +| E. Quota and budget exhaustion | RC2 | RC4 | FR-35 to FR-39 | +| C. Merged fix is not what runs | RC5 | RC7 | FR-45 to FR-53 | +| F. Identity, credential, permission | RC4 | RC3 | FR-26, FR-29, FR-60 | +| G. Re-review churn, treadmill, runaway | RC6 | RC2, RC4 | FR-7, FR-17, FR-20, FR-34 | +| I. Steering does not reach the agent | RC1 | RC4, RC5 | FR-22 to FR-25 | +| J. Tests and evals miss the real thing | RC7 | — | FR-54 to FR-58 | +| K. Automation that corrupts or destroys | RC3 | RC7 | FR-10, FR-14, FR-61 | +| H. Timeout loses the work | RC2 | RC3 | FR-6 | + +### 4.3 The decisions the MVP must make + +| Decision | Closes | Sub-decision | +|---|---|---| +| D-A. Where work-item state lives and who may change it | RC1 | §6.1 | +| D-B. How a run is admitted and queued | RC2 | §6.2 | +| D-C. The run contract and where write credentials sit | RC3 | §6.3 | +| D-D. Which identity an agent acts as | RC4 | §6.4 | +| D-E. The unit of release, and where the factory lives | RC5 | §6.5, §6.6 | +| D-F. What an approval is bound to and who may withdraw it | RC6 | §6.9 | +| D-G. What language holds decision logic, and what blocks a merge or a promotion | RC7 | §6.7, §6.8 | + +The abstractions these decisions introduce are not speculative. Each already exists three or more +times in the POC: the read/write split in the persona runner, issue triage and the planner's +applier (§3.7); seven hold checks; nine "is this actor automation" tests; eleven numeric caps (for example `scripts/dev-lead-fix-issue.sh:35`, +`.github/workflows/pr-review.yml:252`, `scripts/lib/pr-automation-budget.sh:33`, against a standard +that says it "introduces no new count-based circuit breakers", `docs/agentic-interaction-model.md:547`). The rule of three is met +several times over. What is missing is one copy. + +--- + +## 5. Whole-system options + +### 5.1 The four families + +**O1. Keep hardening the current design.** Continue issue by issue. Add the missing guards, make the +existing tests required, fix the budget feed. + +**O2. Restructure on GitHub Actions.** Keep Actions as the only execution substrate. Introduce a +declared lifecycle, one admission point that reconciles state rather than relaying events, one run +contract with the write credential outside the model, one agent identity, one release. No always-on +component. + +**O3. Move the control plane off Actions.** An org-level GitHub App receives webhooks at a hosted +endpoint and orchestrates. Variant O3a keeps the workers on Actions and hosts only the receiver and +queue. Variant O3b adds a durable workflow engine and runs the workers off Actions too. ADR-0007 +weighed both and rejected them "at the current fleet size". + +**O4. Adopt an existing agent platform.** Candidates: `gh-aw` (pinned here at v0.74.4 and only +compile-checked, `.github/workflows/ci.yml:112-118`), `anthropics/claude-code-action` (used by the +planner and idea triage), the Claude Agent SDK, and GitHub's own assignable agents. + +**O4 is not a whole-system option, and is evaluated as a component choice.** Every candidate is a +worker runtime. None declares a work-item lifecycle, admits runs against a budget, binds an approval +to content or defines a release unit, so none touches RC1, RC2, RC5 or RC6. O4 is therefore folded +into decision D-C (§6.3): who builds the harness. It can combine with O2 or with O3. + +### 5.2 Against the requirement groups + +"Yes" means the option meets the group by construction. "Partly" means some requirements can be met +by further local work. "No" means a requirement in the group is a structural property the option +does not have. + +| PRD group | O1 harden | O2 restructure on Actions | O3a receiver + Actions workers | O3b durable engine, workers off Actions | +|---|---|---|---|---| +| §6.1 Lifecycle (FR-1 to FR-7) | **No.** FR-1 to FR-3 need a declared lifecycle with one evaluator; thirty entry points each keep their own `if:`. FR-5 to FR-7 can be patched one loss mode at a time, which is the current practice. | Yes. | Yes. | Yes. | +| §6.2 Run contract (FR-8 to FR-14) | Partly. FR-10 needs the harness rewritten whichever option is chosen. | Yes. | Yes; same harness. | Yes, after rebuilding sandbox, logs, timeouts and artifacts. | +| §6.3 Orchestration (FR-15 to FR-21) | **No.** One admission point cannot be retrofitted onto workflows that each start their own agent. | Yes, with a latency floor: a wake costs a runner start, and a lost wake waits for the next event or sweep. | Yes; real queue and timers. | Yes; strongest. | +| §6.4 Human control (FR-22 to FR-28) | Partly. Identity can change; one vocabulary across thirty workflows cannot be held. | Yes. | Yes. | Yes. | +| §6.5 Review and merge (FR-29 to FR-34) | Partly. Each gate has been patched in turn and the next then failed (§2.4). | Yes. | Yes. | Yes. | +| §6.6 Resource governance (FR-35 to FR-39) | Partly. | Yes for GitHub quota and admission. The LLM signal stays an estimate (§6.2). | Same; the missing signal is constraint C-3, not the substrate. | Same. | +| §6.7 Observability (FR-40 to FR-44) | **No** ledger; reports stay reconstructions. | Yes; a git-backed ledger, queryable with simple tooling. | Yes; a database, better queries, private by default. | Yes. | +| §6.8 Release (FR-45 to FR-53) | **No.** Two regimes and per-agent names are structural. | Yes, except that an entry shim must run from the default branch (§10, item 2). | Worse on FR-46: the receiver is a second deployable with its own release path. | Worse again: three deployables. | +| §6.9 Assurance (FR-54 to FR-58) | Partly. Checks can be made required; portability and unit-callability stay as they are. | Yes. | Yes, plus the hosted service needs its own tests. | Same. | +| §6.10 Security (FR-60 to FR-63) | Partly. | Yes, with one stated exception for the reviewer credential (§6.4). | A second secret store and rotation path (ADR-0007, Context, fact 3). | Same, and the LLM credential leaves Actions. | + +### 5.3 Against the non-functional requirements + +| NFR | O1 | O2 | O3 | +|---|---|---|---| +| NFR-1 flow efficiency | No. Fan-in per event is the design: one comment starts three agentic workflows (`.github/workflows/dev-lead.yml:42`, `persona-mention.yml:42`, `pr-review-mention.yml:31`) and one review starts four. | Runs per merge fall by an order of magnitude once one wake replaces three or four listeners and cross-PR fan-out stops. The first draft's cap on cancelled runs was the wrong measure (§10, item 3). | Best: events that need no work start no run at all. | +| NFR-2 lead time | Flat or worse: median rose from about 1 h to 12–22 h while fixes landed (PRD §2). | Plausible. The floor is runner start plus sweep interval, minutes not hours. | Best. | +| NFR-3, NFR-4 autonomy and gate integrity | No evidence of convergence: 20 of 29 agent merges in the latest week lacked a standing approval (PRD §2). | Yes if §6.9 holds. | Same as O2; these depend on the kernel, not the substrate. | +| NFR-5 liveness | Compensators only. | Yes: every non-terminal state carries a next evaluation time and one sweep reads it. | Yes: durable timers. | +| NFR-8 release lag | No. | Yes. | Yes for the kernel; the receiver has its own lag. | +| NFR-9 operator load | Rising. | Higher during migration, then lower. | Adds a service with no on-call. | +| NFR-11 changeability | No. | Yes. | Yes. | +| NFR-12, NFR-13 | No without leaving bash for decisions. | Yes (§6.7). | Yes. | +| NFR-14 quota headroom | No while agents share the owner's quota. | Likely; depends on reads per evaluation, which is unmeasured (§11). | Likely better: webhook payloads replace reads. | + +### 5.4 Against the constraints + +| Constraint | O1 | O2 | O3a | O3b | O4 component (`gh-aw` as worker) | +|---|---|---|---|---|---| +| C-1 one maintainer, fail to a hold | Load rising: defect inflow 21–36 a week against flat throughput. | Migration adds load for some months. Fails to "nothing happens". | A service to run with no on-call. It also fails to "nothing happens", but silently and for every repo. | Same, plus workers. | A fast-moving dependency to track. | +| C-2 free plan, all public | Fine. | Fine. | Needs a host; free tiers exist but are a new dependency. | Same. | Fine. | +| C-3 subscription capacity, no usage API | Fine. | Fine. | Fine while workers stay on Actions. | The subscription credential must live on a host. | **Conflict as documented.** The `gh-aw` engine reference lists an API key or workload identity for the Claude engine and no subscription token, and states that cross-provider chains fail compilation. Unverified beyond the docs. | +| C-4 no always-on infrastructure | Fine. | Fine. | **Violates** unless the maintainer lifts it (OQ-1). | **Violates.** | Fine. | +| C-5 the platform loses events | Compensated, not tolerated. | Tolerated by design: no event carries work. | Mostly escaped, but GitHub does not retry a failed webhook delivery on its own, so a reconciling sweep is still required. | Same. | No effect. | +| C-6 identity rules | Independent of the option. The identity decision (§6.4) is the same in all of them. | | | | | +| C-7 self-hosting | Fine. | Fine: the hosting repo is served by the promoted release. | The receiver's deploy is outside the repo's own review path unless Actions deploys it. | Same. | Fine. | +| C-8 unreliable advisory reviewers | Patched per bot. | Deadlines in the kernel. | Same. | Same. | No effect. | + +### 5.5 Migration cost, operating load, reversibility + +| | O1 | O2 | O3a | O3b | +|---|---|---|---|---| +| Migration cost | None up front. Unbounded over time: 31% of defect issues already name a prior issue. | High. A kernel, a harness and a controller are built while the POC keeps delivering. Six cutovers. | O2's cost plus a receiver, a queue, deduplication and replay, a second secret store. | O3a's cost plus rebuilding logs, timeouts, artifacts and the sandbox. | +| Risk to the live line | The risk is the status quo. | Contained by running read-only first and cutting over one repo and one intent family at a time. | As O2, plus a new single point for the whole org. | Highest. | +| Maintainer load after migration | Rising. | Lowest that respects C-4: nothing new to operate. | A hosted component to monitor, patch and pay attention to. | More. | +| Reversibility | n/a | High. Each cutover is a registry entry and one consumer file; the POC path stays until retired. | Medium. The App's webhook URL is one setting, but state has moved off GitHub. | Low. | +| What it forecloses | Nothing, and it buys nothing structural. | Nothing. A kernel with no I/O (NFR-12) runs behind a webhook receiver as well as behind a workflow. **O2 is a strict prefix of O3.** | O1. | O1 and most of O2's substrate. | + +### 5.6 Reading the comparison + +- **O1 has earned real results and has stopped converging.** Hard failures are 0.9% of runs. But + eight fixes moved a failure rather than removing it (§2.4), and FR-1, FR-15, FR-45 and + FR-10 are structural properties that no further local guard produces. O1 remains the right choice + under one condition, stated in §7.5. +- **O3 answers a question ADR-0007 did not ask.** ADR-0007 weighed the *ingress file count* and named + three reopen triggers: stub fan-out becoming the bottleneck, events Actions cannot subscribe to, + and cross-org routing. The first has partly fired (a new trigger took 32 days across two repos, + PRD NFR-11); the other two have not. The new evidence is different: Actions is being used as a *queue + and a timer*, and it is poor at both. O3 fixes that directly. But the fix that matters, treating + events as hints and reconciling from recorded state, is needed under O3 as well, and it can be + built without a host. So the order is settled even if the destination is not: build the + reconciling kernel first, on Actions, and reopen the host question with measurements. +- **The identity, release, approval and assurance decisions do not depend on the substrate.** Four + of the seven causes are closed identically under O2 and O3. + +--- + +## 6. Sub-decisions + +Each has options, a recommendation and the condition that would change it. + +### 6.1 Where work-item state lives and who may change it + +| Option | For | Against | +|---|---|---| +| a. Keep deriving state from labels and markers on every run | No migration | RC1 itself. A marker can outlive the object it describes (`scripts/lib/standing-approval.sh:2-15`). | +| b. One machine-written state comment per work item, authoritative | Visible on the item; no second store; FR-2 for free | Still parsed from comment text; no history, no migration path (`scripts/backfill-review-markers.sh` exists for that reason); 65,536-character limit; every read is an API call | +| c. A git-backed store: one state file per work item plus append-only ledger events, committed together; the item shows a projection | Atomic multi-record transitions; history and replay for free (FR-41, FR-58); reads move off the REST quota onto git; schema migration is a script | A push per transition; contention on one ref; the projection can lag; public if kept in a public repo | +| d. GitHub Projects fields | Visible board | Human-editable, no history, GraphQL-only; a projection at best | +| e. An external database | Best queries; private | Only with O3; violates C-4 | + +**Recommendation: (c), with the item showing a projection.** The State record and the Ledger live in +one git store written only by the Controller identity. The Kernel folds ledger events into the State +record; each transition is one commit that adds the events and updates the record. The work item +carries one status block (and, on pull requests, one check run) generated from the record. Labels +remain the *human input* vocabulary (start, hold, release, hands-off); the Kernel reads them as +facts and never as stored state. HTML-comment markers stop being state: new code neither writes nor +reads one for a decision. + +Who may change state: only the Kernel, through a declared transition, executed by the Controller. +A human changes state by making a declared human transition (a label, a review, an Override), which +the Kernel records with the actor. + +The store's location is a follow-on decision. The cheapest start is a dedicated branch of the +hosting repo, which needs no new repo. A private data repo is possible on the free plan and answers +OQ-8 if the maintainer wants cost data private; nothing in the evidence requires it, since the +weekly cost report is already a public issue. + +*Would change it:* push contention under real load (§11, A-10), or a decision for O3, which brings +option (e). + +### 6.2 How runs are admitted and queued + +**A platform fact changed during this review.** The evidence base, the PRD (C-5) and the repo's own +comments all describe a concurrency group as one running plus one pending run. GitHub's +documentation, read 2026-10-10, now describes an optional `concurrency.queue` property: `single` +(the default, the behaviour the repo knows) or `max`, which holds up to 100 pending runs and cannot +be combined with `cancel-in-progress: true`. No workflow in this repo uses it. This is a cheap +mitigation for the POC (§8, step 0) and it does not change the recommendation, for the reason below. + +| Option | For | Against | +|---|---|---| +| a. Status quo: each event starts its own agent; lanes are the queue; crons re-find what was lost | — | RC2 | +| b. Same, with `queue: max` on the lanes | One-line change per lane; removes eviction of pending runs | Does nothing for a dropped tick, a failed dispatch, a killed run, a failure read as "nothing to do", or capacity. Queued runs each still do work. The request still exists only as a run. | +| c. **Level-triggered admission.** Events are wake-ups that carry only the work item's identity. One Controller evaluates recorded state plus current facts and decides: nothing, defer until T, hold, or admit one Run. | A lost or evicted wake loses nothing, because the next wake or the sweep reaches the same decision. The default depth-one lane becomes exactly the coalescing FR-16 asks for. One place evaluates Pause, Hold, budget, duplication and concurrency (FR-15). | A runner start per wake. Timers are only as good as the sweep. | +| d. A hosted queue and scheduler | Real queue semantics and timers | O3 | + +**Recommendation: (c).** Its parts: + +- **Wake.** The consumer's single ingress file (ADR-0007) forwards `{schema version, repo, item}` + and nothing else. The Kernel's decision function takes no event payload, so the property "no + event carries work" is checkable. +- **One lane per work item,** held for the whole of decide → run → apply. That is FR-16 by + construction. An evicted pending wake is harmless. +- **Global limits by compare-and-swap.** Admitting a Run is a commit to the ledger that takes a + lease. A push that loses the race is rebased and the decision is re-made against the new state. + That gives an org-wide concurrency and budget check without an org-wide lock. +- **Timers as data.** Every non-terminal, non-held state records a next evaluation time. A sweep on + two or more staggered cron slots wakes overdue items, and every Controller run does the same + opportunistically, so ordinary activity drives the timers when ticks are dropped. This one sweep + replaces the retry, sweep, delayed-retry and stall-detection workflows of §3.2. +- **Hub execution.** The Controller runs in the hosting repo, woken by `repository_dispatch`. This + is already how personas, the mention path of pr-review, the planner and idea triage work + (`DG:.github/workflows/persona-mention-reusable.yml:365-366`, + `DG:.github/workflows/pr-review-mention-reusable.yml:207-208`); dev-lead is the exception. It gives one ring registry, one ledger writer and a consumer + file that never needs editing. Its cost is in §7.5. +- **Pause and Hold.** Read at admission, in one place. A human Pause is an Actions variable; an + automatic pause is a ledger record. They are different stores, so automation cannot clear a human + Pause (FR-24). The ingress subscribes to `unlabeled`, so releasing a Hold is an event (FR-23). +- **Capacity signal (OQ-3).** GitHub quota: the Controller reads its own installation's remaining + limit, which is exact. LLM budget: the evidence does not support a strong recommendation. The + usage endpoint is unsupported and refuses the CI credential + (`petry-projects/.github-private#2179 — Budget poller: record the upstream error type on a refused + read, so an HTTP 403 says why (every poll so far is degraded)`). The option that needs no new + credential is self-metering: the ledger sums the factory's own token use per window and learns the + ceiling from observed limit hits. It is a lower bound on use, and it is only trustworthy if the + factory's subscription credential is not shared with interactive sessions. *Experiment:* two weeks + of ledger self-metering in observe mode, compared with every observed rate-limit event. + +*Would change it:* see the O3 reopen triggers in §7.5. + +### 6.3 The run contract, and where write credentials sit + +| Option | For | Against | +|---|---|---| +| a. Status quo, with ADR-0011's guards | Exists | Guards relocate the failure: they "refuse legitimate pushes, loop, or retract real fixes" (§2.4) | +| b. One contract, built here: Task in, Result out, jobs as credential boundaries | Generalizes three seams that already work (§3.7); engine-neutral; keeps the subscription credential and cross-provider fallback | It must be built and maintained | +| c. `gh-aw` as the worker runtime | The closest existing implementation of FR-9, FR-10 and FR-12: a read-only agent job, typed safe outputs applied by separate jobs, and outputs that cover this factory's effects (`push-to-pull-request-branch`, `submit-pull-request-review`, `resolve-pull-request-review-thread`, comments, labels). A network firewall and threat detection that option (b) lacks. | Documented Claude auth is an API key or workload identity, not the subscription (C-3). No cross-provider fallback (FR-13; `scripts/engine.sh:2082-2267` exists because the fleet needs one). v0.74 of a fast-moving tool. The repo already declined it once: `scripts/aw.sh` is a 586-line home-grown re-implementation serving one workflow; why is not recorded (§11). | +| d. `anthropics/claude-code-action` as the worker | Already used twice | It is the same CLI in an action wrapper; as used here the model holds the write token (`.github/workflows/initiative-planner.yml:246-256`). It adds nothing to the contract. | +| e. Claude Agent SDK packages | Tool-approval callbacks; native message objects | The headless CLI *is* the Agent SDK's CLI form and already returns what the contract needs: `--output-format json` with cost, usage and session id; `--json-schema` for a validated `structured_output`; `--resume` by session id or transcript path. A package dependency buys little more. | +| f. GitHub's assignable agents | Native | Needs a paid Copilot seat (C-2). Copilot answered 2 of 174 pull requests as a reviewer in one week (`petry-projects/.github-private#1220 — Third-Party Reviewer Scorecard — Weekly Report`, comment of 2026-10-05). | + +**Recommendation: (b), shaped so that (c) can replace its inner half later.** + +- A **Task** is a schema-versioned document containing everything the Run needs, with every piece of + third-party text labelled by trust class. The model does not discover its inputs. +- A **Result** is a schema-versioned document: status, a list of requested **Effects**, usage. Code + changes are local commits handed over as a bundle. The effect vocabulary follows `gh-aw`'s safe + outputs where they overlap, so substitution stays cheap. +- **Three jobs, three credential sets.** `run` holds the LLM credential and a read-only GitHub + token. `verify` holds no secret and is the only job that executes code from the work item's + branch (lint, the repo's tests, the diff guards of ADR-0011). `apply` holds the write credential, + runs no model and no work-item code, validates the Result and applies its Effects. +- **Claims are generated, not verified.** The model never states a commit hash. `apply` writes + "fixed in X" from the commit it pushed and read back. FR-11 then holds by construction, and the + claim-verification, attribution-shim and retraction code (roughly four thousand lines, PRD §6.2) has + nothing left to police. +- **Use what the CLI already returns.** Structured output replaces the brace-scanning verdict + parser (`scripts/engine.sh:2274-2301`); the JSON envelope's cost and usage fields are recorded for + every Run, including failed ones (FR-38); a persisted transcript makes resume possible (FR-6). +- **Residual risk, stated.** The `run` job's model can still read the LLM credential and reach the + network. Option (c)'s firewall closes that; option (b) does not. + +*Would change it, and the experiment:* port issue triage (already front-matter-shaped, +`.github/workflows/issue-triage.md`) to real `gh-aw` on one repo. If the subscription credential +works there and the lack of cross-provider fallback is acceptable for that role, adopt `gh-aw` as the +inner half of `run`/`apply` for tool-less and advisory roles first. + +### 6.4 Agent identity + +**What was decided before, and why.** Interaction model §12.2 kept dev-lead on the owner account and +rejected a distinct identity on four grounds. ADR-0008 then put every advisory persona on the owner +account because the machine user's token could not write comments +(`petry-projects/.github-private#1734 — persona-runner: qa-lead advisory lost — DON_PETRY_BOT_GH_PAT +gets HTTP 403 posting issue comments (regression from #1694)`). Both records are honest about cost: +ADR-0008 names the recursion guard dropping to one axis as "the real cost". + +**Taking §12.2's four grounds in turn:** + +1. *"A distinct identity would need to be added to CODEOWNERS to retain the same approval + semantics."* A pull request's **author** does not need to be a code owner; its **approver** does. + With an App as author, the machine user still satisfies the code-owner review, and the Maintainer + gains the ability to approve or request changes, which the shared identity removes + (`docs/agentic-interaction-model.md:725-733`). The semantics improve. +2. *"A new account means a new PAT, widening the credential surface the model deliberately kept to + two accounts."* True of a second machine user, which is also barred: GitHub's terms allow one + free machine account per person and the org has it. It is not true of a GitHub App. An App has + no PAT; it mints a token per run that expires within the hour and can be narrowed per role. The + org already operates GitHub Apps for release tagging, Dependabot and project sync + (`AGENTS.md:939-951`; §3.4). The agent identities stay at two: one App + that writes, one machine user that reviews. +3. *"It is a cross-cutting change to the manifest and identity-resolution machinery."* True, and it + is the real cost. It is paid once. +4. *"It does not generalize: other automation still posts as the owner, so the marker is needed + regardless."* True only while other automation stays on the owner account. The PRD's exit + criterion 5 removes all of it. §12.2 itself notes that the marker gate "does not foreclose" + a distinct identity and would become "a redundant confirmation". + +**What has changed since.** The costs §12.2 and ADR-0008 did not have in front of them: the loop +ceiling cannot count dev-lead's actions (`scripts/lib/pr-automation-budget.sh:54-59` treats any +login ending in `[bot]` as automation, which an App satisfies with no code change); the Maintainer's +mentions are skipped as self-comments (`petry-projects/.github-private#2027 — dev-lead silently +ignores @dev-lead mentions from the maintainer, because the maintainer's account is dev-lead's own +identity (self-comment skip)`); the shared quota ran out; and no measurement can tell a human action from an agent's (§2.2). + +| Option | FR-26 | FR-29, FR-30 | FR-37 | FR-60 | C-6 | +|---|---|---|---|---|---| +| a. Status quo (owner + one machine user) | No | Reviewer differs; owner cannot review | No | No | Fits | +| b. A second machine user for the writer | Yes | Yes | Yes, a second 5,000/h | Long-lived PAT | **Barred** by the one-free-machine-account term | +| c. **One factory App writes; the machine user reviews** | Yes at agent-versus-human grain | Yes, and enforced by the platform: with code-owner review required, the App cannot satisfy the gate it is subject to | Yes: an installation has its own limit of at least 5,000/h | Yes for the App; not for the reviewer | Fits | +| d. One App per role | Yes at role grain | Yes | Yes | Yes | Fits; N keys and installs to run | +| e. App writes and a second App reviews; drop code-owner review | Yes | Depends on an unverified platform fact (§11, A-1) and removes the platform's own enforcement | Yes | Yes | Needs OQ-4 answered "drop" | + +**Recommendation: (c).** Per-role least privilege comes from narrowing the installation token at +mint time, not from more Apps. The role of each action is a field in the ledger and a line in the +visible output. The marker stays as a redundant guard. + +**Answers to the open questions.** OQ-2: yes, move the writer off the owner account; this +supersedes ADR-0008 and reverses interaction model §12.2. OQ-4: keep code-owner review. It is what +makes reviewer independence a platform fact rather than a factory promise, and the price is that the +reviewer stays a user account with a long-lived token. That token moves into the `apply` job of the +review intent only, where no model runs. It is the one stated exception to FR-60. + +**The cost that must not be buried.** Events authored with an App token start workflows; events +authored with the default workflow token mostly do not. ADR-0007 called the latter a free loop +brake. dev-lead does not have that brake today either, because it already acts through a PAT, but +with this decision nothing has it. Admission (§6.2) becomes the only loop bound. That is acceptable +only because the Kernel's count of automated actions since the last human decision no longer depends +on which account acted (FR-17). + +**One credential the table does not show.** The consumer's ingress must send a wake to the hub, and +it runs in a repo whose branches agents write to. That credential should be able to do one thing: +start the entry workflow. It should not be the App key that writes code. A second, dispatch-only +credential, or a wake sent as a `workflow_dispatch` under an App permission limited to Actions, keeps +a compromised consumer workflow from acquiring write access to the org. The identity follow-on ADR +must decide which. + +*Would change it:* a role that needs a permission the shared App should not hold, or a need to +filter by role on the actor field alone. Either reopens (d). + +### 6.5 The unit of release + +| Option | For | Against | +|---|---|---| +| a. Status quo: 17 per-agent names over one whole-repo commit; personas and hub workflows unversioned | Exists | Two regimes; independence is nominal ("they only appeared to", ADR-0006); 225 dev-lead tags and 137 majors with exactly one release each (local tag snapshot, 2026-10-10) | +| b. Per-agent releases made real, with per-agent watched paths | Isolated rollback | Shared code (`scripts/lib`, the engine layer) makes isolation false; multiplies the gate | +| c. **One release line for the factory**: `factory/vX.Y.Z` immutable, `factory/v-next` and `-stable` moving | One fact; every artifact rides it, prompts and manifests included; no sender/receiver skew inside the factory (FR-52); the interface that could "break" is one wake schema | No per-role staging or rollback; a fix to one role waits for the whole gate | +| d. Live at merge for everything | Fast | The 2026-06 state the channel tags were introduced to end | + +**Recommendation: (c), with two channels, not four.** ADR-0002's mechanism survives unchanged: an +immutable tag and a moving channel under a ruleset, moved by an App through one path. What changes +is the name granularity. `ring0` and `ring1` are dropped until there is traffic to fill them +(`petry-projects/.github#1086 — Gate soak is vacuous: dwell/sample count runs from repos that are +not executing the candidate — and dev-lead's 'next' tier is structurally unoccup…`). + +How the hub follows a channel: a `repository_dispatch` or `schedule` run always executes the +default branch's workflow file, and a `uses:` reference cannot be computed. So the entry workflow is +a small frozen shim on the default branch with one static caller job per channel, chosen by a +job-level `if:` on the output of a ring-resolution job that is itself a pinned reusable. The shim +therefore stays a thin caller under ADR-0001, with one non-event predicate added to ADR-0007's +filter rule. Everything else, the Controller workflow included, is read at the release +the shim selected. Inside the release, tooling is checked out at `github.job_workflow_sha`, the +construction the standards repo already uses to remove the double pin +(`DG:.github/workflows/persona-mention-reusable.yml:54-60,129`). + +Ring membership is one file in the release. The hosting repo is always in `stable`, which preserves +the self-hosting rule (`docs/release/versioning.md:81-95`). The `next` ring is one or two low-risk +consumers plus synthetic work items when organic traffic is absent. + +*Would change it:* two roles needing genuinely independent cadence for a reason other than +convenience. None is in evidence. + +### 6.6 Where the factory lives + +Two questions are tangled here: the two-repo split, and the name. + +**The split.** The router is in the standards repo and reads manifests from this repo's `main`; the +ring registry and the promotion engine (`DG:scripts/canary-rollout.sh`, 4,015 lines) are in the +standards repo and govern agents hosted here; the persona schema exists in both and has diverged +(`standards/personas/persona.schema.json`, 353 lines here against 348 there). Whatever else is decided, the +factory must be one tree with one release: Controller, Kernel, Harness, Roles, ring registry and +release tooling together. The standards repo keeps what its own boundary rule says it owns, org-wide +standards and compliance (`DG:AGENTS.md:38-55`), and distributes the one consumer file. + +**The home of that tree.** + +| Option | For | Against | +|---|---|---| +| a. A `factory/` tree in the hosting repo, with its own required check | No new repo (C-4 as decided in June); the seams in §3.7 are reused in place; one cutover fewer; reversible | New and legacy code share a repo and a 1,062-line `AGENTS.md` of legacy doctrine; a release tag still covers legacy files; the name stays misleading | +| b. A new repo now | A physical boundary; required checks from the first commit; a clean tag namespace; `.github-private` can return to its GitHub-defined role | A new repo to bootstrap (two scripts in two repos and three manual vendor or admin steps, `DG:scripts/bootstrap-new-repo.sh:1-45`) at the moment the line is weakest; two migrations at once; needs the "no new repos" rule lifted | +| c. Fold into the standards repo | One repo | Reverses the boundary rule; puts the largest codebase in the policy repo | + +**Recommendation: (a) now, extraction as a separate later decision.** Make the tree self-contained +(nothing under `factory/` imports from outside it) and give it its own `AGENTS.md`. Extraction then +becomes a mechanical subtree split whenever the maintainer wants it. Answer to OQ-5: not yet; decide +at MVP exit, when the legacy code that makes a new repo attractive has been retired anyway. The +evidence does not make a strong case either way, and (a) keeps both open. + +Two smaller points. The repo's own instructions call it private while it is public +(`.github/workflows/pr-review.yml:358`); correct the instructions now, independent of everything +else. And GitHub gives the `.github-private` name a meaning for org-level agent profiles, so the repo +persists whichever option is chosen. + +### 6.7 Language and runtime for decision logic + +ADR-0004 governs: pure classifiers in shell, a matching bats file, the model only where it must be. +Its principle has held up and is kept. Its language binding is what the evidence questions: + +- control flow rides on overloaded exit codes (2, 55, 100, 101, 124, 3, 4; + `scripts/review-batch.sh:317-323`), and workflows then flatten them + (`.github/workflows/dev-lead-reusable.yml:479-481`); +- subshell semantics are "a recurring bug source … worked around case by case" + (`scripts/engine.sh:1431-1433,1556-1561`), including a wrong model attribution fixed in one tier + and left in two; +- structured data is handled by 1,143 `jq` call sites and 54 GraphQL strings, and many test stubs + return the post-`jq` result, so the real call is never exercised (`tests/dev-lead/unit/test_dev_lead_retry.bats:21`); +- the suite needs bash 5 and GNU tools and fails on the maintainer's machine (`AGENTS.md:174-181`); +- 17 functions exceed 150 lines and two of the largest decision paths are not functions at all. + +A lifecycle with typed states, schema-validated documents and replay of recorded work items is a +poor fit for that medium. + +| Option | For | Against | +|---|---|---| +| a. Stay in bash, enforce the discipline | No rewrite; 70k lines of tests keep their value | Every item in the list above | +| b. **Python 3, standard library only for the Kernel** | Already the repo's second language (four validators, a pytest suite under `tests/dev-lead/integration`); present on runners and on macOS; no dependency stream for a factory whose pull-request pipeline is the scarce resource; dataclasses, enums and a type checker suit a state machine | A second language to hold in mind; no compile step, so type checking must be a required check | +| c. TypeScript | Typed; the Actions ecosystem's native language | A dependency tree and a build; Dependabot traffic; nothing in the repo uses it | +| d. Go | One binary; strong typing | A build-and-publish step becomes part of the release; nothing in the repo uses it | + +**Recommendation: (b).** New decision logic is Python with no I/O in the Kernel. Bash is kept as +step glue of a few lines. The existing scripts are not ported wholesale: the Harness first wraps the +seams worth keeping (the engine tier primitives, push safety, the diff guards) behind ports, and +each legacy decision is re-specified in the Kernel, proven against recorded real work items, then +deleted. + +The evidence separates bash from the alternatives clearly. It does not separate Python from +TypeScript; that choice rests on what is already in the repo and on avoiding a dependency stream. +*Would change it:* a harness dependency that exists only for Node. + +**The cost is real.** The bats suite encodes incident history: `scripts/dev-lead-fix-reviews.sh` +cites 48 issues. Some of that knowledge will not survive translation. The mitigation is §8's observe +step: the Kernel's decisions are compared with the POC's on live traffic before anything is cut +over, and the disagreement list is the migration backlog. + +### 6.8 What blocks a merge, and what gates a promotion + +**Merge-blocking.** No ADR says a test must block anything, and none does. + +- A new required check, `factory / verify`, from the first commit of the tree: unit tests, replay of + recorded work items, schema validation, the lifecycle checks of FR-57, the type check and the + import rule that keeps the Kernel free of I/O. +- A required check must be **hermetic**: it reads nothing outside the merge commit. The validators + that fetch from the standards repo at run time once "red-lined the whole fleet 21 seconds after it + merged" (`personas/validate-personas.py:59-63`); they stay advisory until they are hermetic. +- A required check must **always report**: no `paths:` filter on its workflow, and a `merge_group` + trigger, which the five current required checks already have. +- The legacy bats suite becomes required once it has been green on `main` for a set period. Until + then requiring it would block the line on known-red jobs (`.github/workflows/lint.yml:399-402`). + +**Promotion.** A candidate advances from `next` to `stable` on three records in the ledger, not on +run conclusions: an eval result for every production Intent whose inputs changed; verified Effects +from the `next` ring (a Run of the candidate whose Effects were read back); and no regression in the +ring's integrity counters. A ring with no Runs of the candidate is not evidence; synthetic work +items fill the gap. + +Two limits the PRD should absorb (§10): each eval case is a real model call against the same scarce +window (C-3), so candidates must be batched to about one a day and eval results cached by the hash +of the prompt and harness path; and a "held-out" set in a public repo is held out by convention only. + +### 6.9 Approval durability and re-review cost, with the merge queue on + +Live settings: one approval, code-owner review, last-push approval, stale approvals dismissed on +push, thread resolution, loose status checks, and a merge queue that squashes one entry at a time. +GitHub dismisses an approval when the diff changes *or the merge base changes*. + +| Option | For | Against | +|---|---|---| +| a. Status quo | — | RC6 | +| b. Turn off dismiss-on-push; let the factory withdraw approvals | No platform dismissals; no re-approval step | A factory failure then merges unreviewed content. It fails to an action, not to a hold (C-1). | +| c. **Keep the platform's dismissal; bind the factory's verdict to content; re-issue cheaply when content is unchanged; stop updating branches** | Fails closed. The mechanism exists in miniature (`scripts/lib/carry-forward.sh`). | A defect in the content identity is an unreviewed merge; SM-C3 exists to watch for it. | + +**Recommendation: (c).** + +- **Stop the fan-out at its source.** The queue builds and tests the real merge result, so a branch + does not need to be current. Auto-rebase should act only on a pull request that conflicts. This is + open as `petry-projects/.github-private#1881 — auto-rebase still merges main into every open PR on + every push, though the ruleset no longer requires up-to-date branches — the treadmill's driver + out…` and needs no new structure. +- **A verdict is bound to a content identity:** a hash of the pull request's own diff against its + merge base. When a push changes the head but not that identity, the Harness re-issues the approval + with no model Run. Any doubt (a conflict resolution, an unparseable comparison) means a review. +- **One gate function.** The twelve sequential gates become one Kernel function from facts to + approve or withhold-with-reason. Findings and their dispositions are records in the State record, + keyed by finding and content identity, not minimized-comment state and marker text. +- **Withdrawal has two sources only:** the platform, on new content, and the Kernel, on a new + unresolved finding, as a recorded transition that also removes the pull request from the queue. + The review Role never dismisses its own approval. +- **Advisory reviewers have deadlines** and the awaited set is derived from the reviewers observed + active on that repo (FR-32). +- **An Override is a record.** A merge without a standing independent approval remains possible + through GitHub's own bypass. The Kernel detects it and requires a reason on the item; SM-5 counts + the ones without. + +--- + +## 7. Recommended target architecture + +Sections 7 and 8 are the recommendation as first made. §14 challenged it, and §14.7 lists what +changed in ADR-0013 as a result. §18 changes where it is built: in a new repo, not as a tree in +this one, which replaces the recommendation of §6.6 and the structure shown in §7.3. Where they +differ, the ADR is current. + +### 7.1 The recommendation + +**O2: restructure on GitHub Actions around a declared control plane, one run contract and one +release**, with the identity decision of §6.4 and the worker built here to a contract that an +adopted runtime can later fill (§6.3). + +```text +consumer repo hosting repo (the hub) +───────────── ────────────────────── +any subscribed event factory-entry.yml (default branch, frozen, no logic) + │ on: repository_dispatch [factory-wake], schedule + ▼ resolve ring ──► one static caller job per channel +agent-ingress.yml (ADR-0007) │ + one job, event filter only ▼ + sends {v, repo, item} ───────────────► factory-controller.yml @ factory/v- + one lane per work item, held for the whole run + ┌────────┐ facts + State record ─► Kernel ─► Admission + │ decide │ nothing │ defer(T) │ hold │ admit(Run) + └───┬────┘ one ledger commit = the decision + a lease + ▼ + ┌────────┐ Task ─► Engine ─► Result + │ run │ LLM credential, read-only GitHub token + └───┬────┘ + ▼ + ┌────────┐ the only job that executes work-item code + │ verify │ no secrets + └───┬────┘ + ▼ + ┌────────┐ validate Result ─► apply Effects ─► read back + │ apply │ write credential; no model, no work-item code + └───┬────┘ ledger commit ─► projection on the item + ▼ + git-backed Ledger + State records (append-only, Controller-written) +``` + +| Decision | Recommendation | +|---|---| +| State | Declared Lifecycle as data; State record and Ledger in one git store; labels are inputs; markers are not state | +| Admission | Level-triggered Controller in the hub; wakes carry identity only; one lane per work item; leases by compare-and-swap; timers as data with one sweep | +| Run contract | Task in, Result and Effects out; `run` / `verify` / `apply` are separate credential boundaries; claims generated by the Harness | +| Identity | One factory App writes; the existing machine user reviews; the owner account is the human's alone | +| Release | One line, `factory/`, two channels, everything on it; a frozen entry shim is the only thing live at merge | +| Home | A self-contained `factory/` tree in the hosting repo; extraction decided at exit | +| Language | Python 3, standard library only in the Kernel; bash as glue | +| Assurance | `factory / verify` required and hermetic from the first commit; promotion on eval and ring records in the ledger | +| Approval | Platform dismissal kept; verdict bound to content identity; one gate function; auto-rebase only on conflict | + +### 7.2 Implementation patterns + +These are the rules that keep many authors, most of them agents, producing compatible code. + +1. **Facts in, decision out.** A Kernel function takes plain data and returns plain data. It reads + no environment, file, clock or network; the clock is an argument. +2. **Events are hints.** No decision reads an event payload beyond the work item's identity. +3. **Effects are data.** A Role requests; only `apply` acts. A new kind of side effect is a new + entry in the effect schema, never a new credential in a new place. +4. **A job is a credential boundary.** No job holds both an LLM credential and a write credential. + No job that holds a write credential executes work-item code. +5. **One writer per work item,** by the lane; one writer of the ledger identity, by construction. +6. **Every document is schema-versioned:** wake, Task, Result, Effect, State record, ledger event, + Role manifest, Lifecycle. A reader rejects a version it does not know. +7. **Derive, never enumerate.** The set of Roles is the directory listing; the consumers and rings + are one file; a table in a document is generated and checked for staleness. +8. **Fail to a Hold.** An unreadable fact, a missing capacity reading or a failed validation yields + a Hold or a deferral with a reason. It never yields "nothing to do" and never an action. +9. **Content identity, not commit identity,** decides whether work is new. +10. **No marker is state.** New code does not read a comment body to decide anything. The one + adapter that reads legacy markers during migration is on an allowlist that only shrinks. +11. **Workflow YAML carries no logic.** A step calls a module entry point; a `run:` block is a few + lines. +12. **A required check is hermetic and always reports.** + +### 7.3 Structure + +```text +factory/ + AGENTS.md rules for changing the factory; supersedes legacy doctrine for this tree + lifecycle/ one file per work-item type: states, transitions, gates, actors (FR-1) + roles// manifest, prompts/, evals/ (FR-8) + schemas/ wake, task, result, effect, state-record, ledger-event, manifest + config/ rings (consumers and ring), tiers (engine per tier), capacity, reviewers + kernel/ pure decision logic: lifecycle, admission, gates, content identity, + dispositions, capacity, projection (NFR-12) + adapters/ all I/O: github (facts, effects), ledger (git), engine (per provider) + harness/ build Task, run Engine, validate Result, apply Effects + controller/ entry point: wake → decide → record → run → verify → apply + release/ cut, promote, roll back; ring evidence from the ledger + tests/ unit/ replay/ (recorded work items) contract/ (recorded GitHub and + engine responses) lifecycle/ (termination, cycles, hold and pause) +.github/workflows/ + factory-entry.yml the frozen shim + factory-controller.yml the only workflow that invokes an Engine + factory-verify.yml the required check +templates/agent-ingress.yml the one consumer file +``` + +| PRD group | Lives in | +|---|---| +| §6.1 Lifecycle | `lifecycle/`, `kernel/lifecycle`, `adapters/ledger` | +| §6.2 Roles and run contract | `roles/`, `schemas/`, `harness/`, `adapters/engine` | +| §6.3 Orchestration | `kernel/admission`, `controller/`, the two factory workflows | +| §6.4 Human control | `lifecycle/` (human transitions), `kernel/admission` (Pause, Hold) | +| §6.5 Review and merge | `kernel/gates`, `kernel/content identity`, `kernel/dispositions` | +| §6.6 Resource governance | `kernel/capacity`, `config/capacity`, ledger leases | +| §6.7 Observability | `adapters/ledger`, `kernel/projection`, scorecard queries | +| §6.8 Release | `release/`, `config/rings`, `factory-entry.yml`, the consumer template | +| §6.9 Assurance | `tests/`, `factory-verify.yml`, `roles/*/evals` | +| §6.10 Security | the job boundaries in `factory-controller.yml`, token narrowing in `adapters/github` | + +### 7.4 Validation against the requirements + +**Coherence.** The decisions depend on one another in one direction only. Level-triggered admission +needs a State record; content-bound approval needs the single gate function; the App identity makes +the loop bound countable, and the loop bound is what makes the App identity safe. One release makes +schema skew inside the factory impossible, which is why the wake can be the only versioned +interface. Nothing in the set contradicts another member. + +**Coverage.** Every requirement group has a home (§7.3). Requirements this architecture meets only +with a stated qualification: FR-2, FR-15, FR-35, FR-46, FR-51, FR-55, FR-60, NFR-1 (§10). + +**Gaps that remain.** + +- *Important:* the LLM capacity signal is an estimate (§6.2). The `run` job has no network + containment (§6.3). Reads per evaluation, and so NFR-14, are unmeasured. +- *Deferred by the PRD's own scope:* the planning roles stay on their current runtime, with the + write token in the model. Exit criterion 5 nonetheless requires them off the owner account. +- *Not addressed here:* review depth, and the reliability of advisory reviewers beyond a deadline. + +**Readiness.** Ready to begin with the first two steps of §8. Not ready to be costed: OQ-9 cannot be +answered from this evidence. + +### 7.5 The trade-offs, stated plainly + +- **It keeps the substrate that caused the second-largest failure class.** The recommendation bets + that reconciling from recorded state makes dropped ticks, evicted runs and slow queues harmless + rather than absent. If the bet is wrong, O3 is still to be bought. This is the strongest argument + against the recommendation. +- **It is a rewrite of the core, in a second language, built largely by the agents it replaces, on + a line that is already overloaded.** Two systems coexist for months and the maintainer's load + rises before it falls. The pattern "a fix relocates the failure" can recur at the scale of an + architecture. +- **Hub execution concentrates risk.** A bad `stable` release or a broken entry shim stops every + repo at once. ADR-0007 accepted that blast radius per repo; this accepts it per org. +- **A consumer's work runs somewhere else.** Runs for a consumer's work items execute and log in the + hosting repo. The consumer's own Actions tab shows only the wake; the status block on the item is + the link. `verify` executes the consumer's tests on a hub runner, so each consumer must declare how + its environment is set up and how its tests are run, which the POC infers. +- **One release removes per-role staging and rollback.** ADR-0006 made the same trade for personas + and observed that the isolation it gave up was never real. +- **A git-backed ledger is not a database.** It will be slow to query at volume and contended under + bursts, and by default it is public. +- **Fewer rings means less staged exposure** than the configured four, though more than the four + delivered in practice: all four channels of dev-lead pointed at one commit at the snapshot, as did + all four of pr-review (§3.5). +- **Re-approval on unchanged content approves a commit no reviewer read.** + +**Choose differently when:** + +| Condition | Then | +|---|---| +| The maintainer cannot fund a migration (OQ-9) or will not pause feature work on the POC | Do step 0 and the identity cutover of §8 and stop. That is O1 with its three cheapest structural fixes. RC1, RC2, RC3 and RC5 remain. | +| OQ-1 is answered "always-on infrastructure is acceptable" **and** a reopen trigger below has fired | O3a, hosting the same Kernel. | +| `gh-aw` is shown to run on the subscription credential and reaches a stable release line | Adopt it as the inner half of `run`/`apply`, advisory roles first. | +| LLM capacity moves to a metered API (C-3 changes) | The capacity signal becomes exact, and the `gh-aw` conflict disappears. Revisit §6.2 and §6.3. | + +**Reopen triggers for the rejected options.** + +- *O3 (off-Actions control plane).* After the review and fix intents are under the Controller: more + than 5% of **admitted Runs** are lost or start later than the liveness limit for four weeks; or + the hub's run queue is the measured bottleneck on lead time; or GitHub quota cannot be held under + NFR-14 with reconciling reads; or a requirement appears for a timer finer than the sweep can give. + These replace ADR-0007's file-count triggers, which the single wake makes moot. +- *O4 as worker (`gh-aw`).* The experiment in §6.3 succeeds, or a prompt-injection incident occurs + that network containment would have stopped. +- *Per-role Apps.* §6.4. +- *A new repo.* MVP exit, or the legacy tree's rules measurably misleading agents that work in + `factory/`. +- *O1.* Never as a strategy. As a fallback, the first row of the table above. + +--- + +## 8. Migration shape + +An order, not a plan. Each step leaves the line running and can stop there. + +**The rule throughout:** for any repo and intent family, exactly one line serves it, the POC or the +factory, as recorded in the ring file. Never both. ADR-0009's dual serving of one surface is the +case to avoid. + +| Step | Introduces | Can be retired when it holds | Evidence that it worked | +|---|---|---|---| +| **0. Floor.** Configuration only; needs no new structure. | Auto-rebase limited to conflicting pull requests. `concurrency.queue: max` tried on the dev-lead and pr-review lanes of one repo. The factory App created and installed, unused. `factory / verify` added as a required check over an empty tree. The repo's instructions corrected to say it is public. The Claude Code CLI version pinned (§11.2, A-8). ADR-0011 accepted as the interim rule. | Nothing yet. | Agent runs on other pull requests per merge to `main` fall to about zero. Cancelled-before-start share on the trial lanes, before and after. | +| **1. Observe.** *The first structural step that delivers value alone.* | The Lifecycle files, the Kernel, the Ledger and the projection, running read-only on every repo under the App's own quota and released through the new single line. It decides nothing and applies only the status block. | After parity: `scripts/pr-approval-diagnostic.sh`; the stall, runaway and merge-ready scans; the premature-closure audit; the zombie report. | For a sample of stuck pull requests the Kernel names the same blocking gate the POC enforced. Undeclared transitions are counted. The scorecard exists, so the PRD's `[ASSUMPTION]` targets can be calibrated (OQ-7). The disagreement list between Kernel and POC is produced. Installation quota use is measured. | +| **2. Identity.** In parallel with step 1; touches POC code. | dev-lead and the personas act through the App; the reviewer is unchanged. The ADR that supersedes ADR-0008 lands with it. One canary repo first. | The marker as *sole* recursion guard; the attribution shim (`scripts/lib/claim-landing.sh:115-170`); the owner PAT on agent paths. | Seven days with no agent action under the owner login. The Maintainer requests changes on an agent pull request and it blocks. The loop ceiling counts dev-lead's actions. Fleet traffic no longer draws on the owner's quota. | +| **3. Review under the Controller.** First on a `next`-ring consumer, then the rest, the hosting repo last. | Admission, the run contract and the single gate function for the review intent; content-bound verdicts; dispositions as records. | `pr-review-sweep`, `pr-review-delayed-retry`, `repair-pr-approvals`, `dismiss-stale-bot-reviews`; the standing-approval, self-dismissed-approval, verify-approval and carry-forward libraries; the three review stubs in each cut-over repo. | No approval dismissed by its own Role. Share of agent pull requests merged with a standing approval on cut-over repos against the rest. Class A inflow. | +| **4. Fix intents under the contract.** | fix-CI, fix-review-findings and resolve-conflicts with no write credential in the model; generated claims. The ADR that supersedes ADR-0011 lands here. | `dev-lead-retry` and its four scans; the `resume` job; the dispatch-channel gate and payload-field check; claim verification, landing and retraction. | No published claim without a pushed commit. Rework share of spend, now measurable. Class B and D inflow. | +| **5. Implementation and the driver under Admission.** | implement-issue and the initiative driver admitted by the Controller; org-wide capacity, priority, ceilings and Pause. | The run-time fetch of the limit gate from another repo's default branch (`scripts/dev-lead-fix-issue.sh:415-452`); the partial pause switch; the dry-run budget poller. | A Pause stops every Role and resumes without re-triggering. A deferred Run shows its reason and time. No Run started against an exhausted window. | +| **6. Personas and one release for everything.** | Advisory Roles on the contract; promotion gated on eval and ring records; the last per-agent channels retired. The ADRs that supersede ADR-0002 and ADR-0006 land here. | The persona runner and the router in the standards repo; the standards repo's promotion engine for factory agents; the stub-drift tools; the hand-written interaction table and its validator. | No manual promotion for four weeks; one rollback exercised; one new advisory Role added in one pull request. | +| **7. Exit.** | The PRD's four-week criteria. The decision on a separate repo. | Whatever in §3.2 still lacks a recorded reason to remain. | §10.3 of the PRD. | + +Notes on the order: + +- **Step 1 before anything that decides.** It tests the Lifecycle against real traffic while being + unable to harm it, and it supplies the measurements whose absence the PRD records. If the + disagreement list is long, that is the finding, and it costs nothing in production. +- **Review before fixing.** The largest failure class is the approval that never issues; the review + role already applies its effects from the wrapper, so its harness change is the smallest; and a + working gate is what makes the later cutovers verifiable. +- **Do not fan out ADR-0007's per-role ingress to the remaining repos first** + (`petry-projects/.github-private#2177 — [Decision] Fan out the ADR-0007 ingress collapse to the + rest of the fleet: go/no-go, repo order, and runbook, using the markets pilot evidence`). Each + repo would then be migrated twice. Let steps 3 to 6 install the one-job ingress as they cut over. +- **The POC is frozen for features from step 1.** Defect fixes only. Otherwise the target moves. + +--- + +## 9. Relationship to ADR-0001 through ADR-0012 + +ADR-0000 offers three statuses and one permitted edit to an accepted record. It has no way to mark +an ADR as extended or narrowed, a gap ADR-0010 recorded and left open. The table uses four words; +only "supersede" changes a status. + +| ADR | Verdict | What happens | How to record it | +|---|---|---|---| +| 0000 process | **Keep.** | ADR-0013 follows it. The missing `extended-by` link is felt again here. | Recommend, separately, an ADR that supersedes 0000 to add a forward link for extension. Not required by this decision. | +| 0001 thin caller / reusable | **Keep, and tighten.** | The consumer file and the entry shim are both thin callers. The boundary gets stricter: the shim has no logic at all. | ADR-0013 cites it; no status change. | +| 0002 channel tags and rings | **Supersede**, restating what survives. | Two tag kinds, ruleset protection and one App-moved path survive. The unit changes from per reusable to the factory; tiers reduce to two. Non-factory reusables in the standards repo keep their own names. | The release follow-on ADR supersedes it at step 6. Until then it governs the POC's tags. | +| 0003 rubric registry | **Keep, narrowed in scope.** | Binding review target to method by data is the same idea as the Role manifest. The registry stays an input adapter above the Engine port. | No status change; ADR-0013 names it as a seam. | +| 0004 pure logic plus bats | **Supersede**, restating the principle. | Purity, I/O at the edge, the model only where it must be, and fail-loud all survive and get stricter. The binding to shell and bats does not. | The language follow-on ADR supersedes it when the first Kernel module merges. Legacy scripts stay under 0004 until deleted. | +| 0005 advisory by default | **Keep, extended.** | Write stays an explicit opt-in. The opt-in is expressed as permitted Effects in the manifest and enforced by `apply`, which also closes the gap that nothing enforces `gate_label` today (§3.6). | The run-contract follow-on ADR extends it; no status change. | +| 0006 shared-runtime personas not ring-registered | **Supersede**, restating what survives. | Its insight, that a persona is data served by a runtime and the runtime is the unit of release, becomes the rule for every Role. Its boundary that the reach check is the sole promotion gate is replaced by the release gate. | The release follow-on ADR supersedes it at step 6. | +| 0007 one ingress per repo | **Keep, extended.** | One file per repo stands. One job per role is replaced by one job, because per-role permissions and pins no longer live in the caller. The rejection of a hosted receiver stands, with new reopen triggers (§7.5). | The admission follow-on ADR extends it, as 0010 did; no status change. | +| 0008 personas post as the owner | **Supersede.** | Reversed, for the reasons in §6.4. Its analysis of the recursion guard is carried forward: the actor axis returns. | The identity follow-on ADR supersedes it at step 2. This is the only reversal of a decision that was argued in full. | +| 0009 qa-lead surface through the router | **Keep until it lapses.** | "No second ingress path" becomes true by construction. Its hard precondition, that a human hold binds on every surface, is met by admission. The router it names is retired at step 6. | Superseded by the admission follow-on ADR when the router is retired; until then it governs. | +| 0010 ingress concurrency; schema is an allowlist | **Keep the principle; the key lapses.** | The allowlist rule is kept and made stricter. Job-level concurrency in the consumer file is no longer needed, because the lane is in the hub. | The admission follow-on ADR extends 0007 again and notes that 0010's key is unused; no status change. | +| 0011 write-mode personas on a shared branch (proposed) | **Accept now; supersede later.** | Its five rules are right for the POC and already implemented. Under the run contract, rules 2 and 3 hold by construction and rules 1, 4 and 5 become `verify` and `apply` steps. | Accept it as the interim record at step 0. The run-contract follow-on ADR supersedes it at step 4. | +| 0012 comment gate's exemption set (proposed; landed on `main` while this review was open) | **Keep for the POC; supersede later.** | It narrows the POC's comment gate by registered marker or one exact body form. It is a further patch to the gate §14.3 identifies as the source of the largest failure class, and it is the right patch for the POC. Under the Finding contract a comment body gates nothing, so the rule has nothing left to exempt. | The merge-integrity follow-on ADR supersedes it when the factory's gate takes over. | + +Two normative documents that are not ADRs also change: interaction model §12.2 is reversed by the +identity decision and should say so with a pointer, and its §§5 to 9 are largely replaced by the +Lifecycle files once those exist. + +**Why ADR-0013 supersedes nothing itself.** It fixes the structure and binds new code. Each +supersession is recorded by the follow-on ADR that lands with the step that makes it true in the +running system. The alternative, flipping several statuses on acceptance, would make the record claim +things the system does not yet do, which is the reverse of the defect ADR-0008 was written to repair. + +--- + +## 10. Where the first draft of the PRD was wrong, over-specified or unachievable + +The analysis tested the PRD against the constraints and found twelve problems. The PRD has been +revised for each. They are kept here because the reasons are architectural and the next requirement +written will meet the same limits. "Achievable" states what the PRD now says. + +1. **FR-2, "the visible State and the State record never disagree."** Two objects on an eventually + consistent platform will disagree for some interval. Achievable: a disagreement is detected and + repaired at the next evaluation, and counted. +2. **FR-46, "nothing is live at merge."** Unachievable on Actions as written. A `schedule` or + `repository_dispatch` run always executes the default branch's workflow file (GitHub + documentation, read 2026-10-10). Achievable: the only artifacts live at merge are frozen entry + shims that contain no logic. +3. **NFR-1.** It counts workflow runs and caps cancelled or evicted runs at 5%. Under level-triggered + admission an evicted wake is the coalescing FR-16 asks for. Count **Runs** per merged change, cap + lost **admitted Runs**, and report wakes separately. +4. **FR-15, "every Admission decision is recorded."** Recording every identical refusal of a held + item is noise. Record a decision when it differs from the last one for that item; count repeats. +5. **FR-35, a "positively verified" capacity signal.** Not achievable for the LLM window under C-3: + there is no supported usage interface. Achievable: an estimate that defers when missing. The + first draft also lacked the LLM counterpart of FR-37: if the factory's subscription credential is + shared with interactive use, the estimate is unsound. FR-37 and OQ-3 now say so. +6. **FR-55, "held-out."** Every repo is public (C-2), so a held-out set is readable by anyone, + including the agent that edits the prompt it scores. Either keep it in a private data repo, which + touches the open half of C-4, or call it a convention guarded by code ownership. +7. **FR-60, "scoped … and short-lived."** Conflicts with C-6 while code-owner review is + required: the reviewer must be a user account. One stated exception, confined to the `apply` job. +8. **FR-51, "fully served within an hour."** Holds for Roles. Advisory reviewers need per-repo + installs in vendor consoles that the factory does not control (`DG:scripts/bootstrap-new-repo.sh:1-45`). And "a new trigger requires + no edit in the consumer" holds only if the one consumer file subscribes to a fixed superset of + events from the start, which the recommendation does. That superset cannot include "any workflow + finished": `workflow_run` must name workflows, so CI completion is heard only for repos whose CI + carries a standard name (finding 6 below); others fall back to the sweep. +9. **FR-26, "attributable by actor alone."** Achievable at the grain of agent versus human. At the + grain of Role it would need one App per Role, which is not recommended. +10. **C-5 was out of date in two places.** One running plus one pending is now the default, not the + limit (`queue: max`). And events authored with the default workflow token do start + `workflow_dispatch` and `repository_dispatch` runs; the repo's two contradictory statements of + this rule (`docs/agentic-interaction-model.md:260-263` against `scripts/auto-rebase-retry.sh:80-81`) + are resolved by the documentation in favour of `scripts/auto-rebase-retry.sh`. +11. **NFR-8 against C-3.** A 48-hour release lag with eval-gated promotion is feasible only if + candidates are batched; six cuts a day, the current cadence, would spend the window on evals. +12. **Exit criterion 5 against §10.2.** "The Maintainer's account authored no agent action" pulls + the planner and idea triage, which the scope leaves "running as they are", into the identity + cutover. §10.1 now says so. + +--- + +## 11. What could not be determined, and the platform facts this rests on + +### 11.1 Verified against official documentation on 2026-10-10 + +| # | Fact | Source | +|---|---|---| +| V-1 | Events authored with the default workflow token create no run, except `workflow_dispatch` and `repository_dispatch`, which always do. An App installation token or a PAT does trigger runs. | docs.github.com, "Triggering a workflow" | +| V-2 | An App installation has its own limit of at least 5,000 requests an hour (more above 20 repos or 20 users, capped at 12,500). The default workflow token has 1,000 an hour per repo. A user has 5,000. | docs.github.com, "Rate limits for the REST API" | +| V-3 | A concurrency group has one running entry. Pending defaults to one (`queue: single`); `queue: max` allows up to 100 and cannot be combined with `cancel-in-progress: true`. Order is not guaranteed. | docs.github.com, "Control the concurrency of workflows and jobs"; "Workflow syntax" | +| V-4 | A code owner is a user or a team with explicit write access. The page does not list Apps. | docs.github.com, "About code owners" | +| V-5 | The merge queue does not require a branch to be current, needs required checks to run on `merge_group`, and removes an entry on failure, timeout, manual removal or an unresolvable protection failure. | docs.github.com, "Managing a merge queue" | +| V-6 | Stale approvals are dismissed when the diff changes or the merge base changes. The last-push rule requires approval from someone other than the last pusher. | docs.github.com, "Available rules for rulesets" | +| V-7 | `schedule` runs can be delayed and, under load, dropped; they run on the default branch. `repository_dispatch` runs the default branch's file; its payload is limited to 10 top-level properties. | docs.github.com, "Events that trigger workflows" | +| V-8 | One person may keep one free machine account besides a personal account. | GitHub Terms of Service, B.3 | +| V-9 | `gh-aw`: a read-only agent job with typed safe outputs applied by separate jobs, including push to a pull-request branch, submit a review and resolve a thread. Claude engine auth is documented as an API key or workload identity. No cross-provider fallback. | github.github.com/gh-aw, safe-outputs and engines references | +| V-10 | Claude Code headless: JSON output carries cost, usage and session id; `--json-schema` returns validated structured output; `--resume` takes a session id or a transcript path. `--bare` mode ignores OAuth credentials and is documented as the future default for `-p`. | code.claude.com/docs, "Run Claude Code programmatically" | + +### 11.2 Assumptions, not verified + +| # | Assumption | Matters for | Cheapest check | +|---|---|---|---| +| A-1 | Whether an App's approval counts toward the required approval count. | Only option (e) of §6.4, which is not recommended. | Not needed. | +| A-2 | An App-authored pull request needs no extra approval. The ruleset page describes one for Copilot-authored pull requests. | §6.4 | One pull request on the canary repo in step 2. | +| A-3 | An App can enable auto-merge and add a pull request to the queue. | §6.4, §6.9 | Same. | +| A-4 | The queue evaluates reviews only on entry. Observed here (`scripts/lib/merge-queue-dequeue.sh:5-8`); the documentation is silent. Whether a push to a queued pull request removes it is also unstated. | §6.9 | Already observed; keep the dequeue. | +| A-5 | An installation token can be minted with a subset of the App's permissions and repositories. Long-standing API behaviour; not re-read today. | FR-60 | Step 0. | +| A-6 | A `uses:` reference to another repo cannot be an expression. The documentation states this for the local form. | §6.5 | A lint run. | +| A-7 | `queue: max` is available to this org and accepted by its linters. | Step 0 | Try it on one lane. | +| A-8 | The subscription token keeps working for `-p` once `--bare` becomes the default. If not, every Role needs an explicit opt-out or C-3 changes. | Everything that runs a model | Watch the changelog; pin the CLI version (the POC resolves `latest` at run time, `.github/workflows/dev-lead-reusable.yml:387-411`). | +| A-9 | A fine-grained token cannot submit a review (`.github/workflows/pr-review.yml:174-178`). | The reviewer's blast radius | Try the REST endpoint with a fine-grained token. | +| A-10 | Ledger pushes do not contend badly at real volume. | §6.1 | Step 1 measures it. | +| A-11 | Reads per evaluation keep the installation under 60% of its limit. Conditional requests are not documented as free. | NFR-14 | Step 1 measures it. | +| A-12 | The free plan's concurrent-job ceiling is not the binding limit once fan-out stops. A queue of 200+ runs was observed on 2026-10-10. | The O3 reopen trigger | Measure after step 0. | + +### 11.3 Not determined + +- **The migration budget (OQ-9).** Nothing in the evidence prices the kernel, the harness or six + cutovers in weeks or in LLM spend. +- **What a full eval pass costs** against the subscription window, and so how often a candidate can + be gated. +- **Why `gh-aw` was re-implemented rather than used.** The reason would bear on §6.3. +- **The logic inside the standards repo's routers,** and what the deployed channel tags run as + opposed to `main`. The review read the hosting repo's tree. +- **Live values of the Actions variables** that override defaults, including the automation-login + list on which the loop-ceiling finding depends. +- **Whether the budget poller has ever produced a usable reading.** +- **Current consumer pins.** The local checkouts are twelve weeks old. +- **How much of the bats suite's incident knowledge can be recovered as recorded fixtures.** Step 1's + disagreement list is the first measurement. + +--- + +## 12. Defects found in passing + +These are not architecture. Each is a concrete fault worth fixing whichever option is chosen. Items +marked ✓ were re-checked by hand against the code or the documentation; the others were read once. + +| # | Defect | Evidence | +|---|---|---| +| 1 ✓ | The per-pull-request automation budget cannot count dev-lead. Its bot list omits the account dev-lead acts as, so dev-lead's activity resets the count. The effect is inferred from the code and was not observed in a run. | `scripts/lib/pr-automation-budget.sh:32`; `personas/dev-lead/persona.yml:107-109` | +| 2 ✓ | `AGENTS_PAUSED` pauses dev-lead only. pr-review's error message tells operators to use it, and pr-review does not read it. | `.github/workflows/dev-lead-reusable.yml:116`; `.github/workflows/pr-review.yml:344` | +| 3 ✓ | The fleet monitor's wake-up for dev-lead carries no pull-request or issue number, and the receiver skips a payload without one. The interaction model cites this as the canonical bridge. | `.github/workflows/actions-fleet-monitor.yml:489`; `scripts/dev-lead-intent.sh:682-686` | +| 4 ✓ | The CI failure analyst's prompt loses its `run_id` variable to unscoped `envsubst`, so the model is told to run `gh run view ""`. | `.github/workflows/ci-failure-analyst-reusable.yml:115-117`, `:159` | +| 5 ✓ | Four bats files run in no workflow: `test_cli_error_routing`, `test_rate_limit`, `test_review_one_pr_escalation_exit`, `test_validate_engines`. `scripts/validate-engines.sh` is wired to nothing. | `.github/workflows/lint.yml:95-258` | +| 6 ✓ | The timeout guardrail checks the sum of all tier budgets against dev-lead's 120-minute job, which runs one tier. The job that chains the tiers is pr-review's, capped at 60 minutes. | `scripts/dev-lead-timeout-guardrail.sh:29`; `.github/workflows/pr-review.yml:157` | +| 7 | Token metering records only successful calls, and names the wrong model after an in-chain fallback in the deep and writer tiers. | `scripts/engine.sh:1801`, `:1962`, `:1740-1745` | +| 8 ✓ | The proactive Claude headroom probe needs `ANTHROPIC_API_KEY`, which no workflow supplies, and logs "usage 0% — ok" on every call. | `scripts/engine.sh:500-517`, `:591` | +| 9 ✓ | `scripts/cut-release.sh` does not know three registered agents: `apply-repo-settings`, `idea-triage`, `initiative-planner`. | `scripts/cut-release.sh:70-89` | +| 10 | The standards repo still dispatches to an `idea-enhancer.yml` that was removed from this repo. | `DG:.github/workflows/idea-enhancer-reusable.yml:71` | +| 11 ✓ | The Claude Code CLI resolves to `latest` at run time unless a variable pins it. Its documentation says bare mode will become the default for `-p` and that bare mode does not read OAuth credentials, which is how every agent here authenticates. | `.github/workflows/dev-lead-reusable.yml:396-401`; code.claude.com/docs, "Run Claude Code programmatically", read 2026-10-10 | +| 12 ✓ | dev-lead fetches a gate script from another repo's default branch at run time and sources it, failing open on any error. | `scripts/dev-lead-fix-issue.sh:404-452` | +| 13 | `release/registry.yml` is a second ring registry that only a test reads, and it disagrees with the real one. | `release/registry.yml:1-41` | +| 14 | Four documents contradict the code on live behaviour: `AGENTS.md:92-93` says dev-lead reads only an issue's title and body while `scripts/dev-lead-fix-issue.sh:573` passes comments; the merge-queue decision record says deferred while the queue is on (`docs/initiatives/auto-rebase-vs-merge-queue.md:3-9`); `CLAUDE.md` calls this repo private; and the interaction model's rule on the default token and dispatch events (`docs/agentic-interaction-model.md:260-263`) contradicts both `scripts/auto-rebase-retry.sh:80-81` and GitHub's documentation. | as cited | +| 15 | The daily fleet failure report appears to have stopped after 2026-09-20, and sampled fleet-monitor runs fail. Not confirmed from run logs. | Issue history; `petry-projects/.github-private#1852 — petry-projects — workflow failures detected 2026-09-20` is the last | + +--- + +## 13. Comparison with other systems + +Added after the options analysis, to test the recommendation against systems built by other +people. Sources were read on 2026-10-10. Each is marked **P** (primary source read directly) or +**S** (secondary coverage; details may be wrong). + +### 13.1 The systems + +| System | Human in the loop | Orchestrator | Where work state lives | How roles interact | Who writes to the outside world | How correctness is established | +|---|---|---|---|---|---|---| +| **StrongDM Software Factory** (P for principles, S for the rest) | None by rule: "Code must not be written by humans. Code must not be reviewed by humans." | A non-interactive agent loop | Not published | Not published | The agent | Scenarios held outside the codebase, "similar to a holdout set", judged by an LLM as a satisfaction fraction; behavioural clones of third-party services | +| **OpenAI Symphony** (P: `SPEC.md`) | Work can end at a handoff state such as human review | A long-running service that ticks every 30 s; reconciliation runs before dispatch on every tick | The issue tracker. Scheduler state is in memory only; restart recovery is "tracker-driven and filesystem-driven (without a durable orchestrator DB)" | Not role-to-role: the orchestrator dispatches one agent per ready issue into an isolated workspace | The agent, through tools; "each implementation defines its own trust boundary" | Left to the workflow; secondary sources describe evidence such as CI status attached to the pull request | +| **OpenAI harness engineering** (S; the primary post could not be fetched) | Reported as optional review | Engineers prompting agents; later Symphony | The repo: a structured `docs/` tree is the system of record | Agents request reviews from other agents and iterate until they are satisfied | The agent | Mechanical architecture rules; background agents that open small cleanup pull requests | +| **Gas Town** (S) | One human talks to one coordinating agent | A local, always-on town of agents | Beads, a ledger that began as JSONL in git plus SQLite and moved to Dolt, a versioned SQL database | Named roles: Mayor decomposes and assigns, Polecats work in isolated worktrees, a Refinery runs a merge queue per project, a Witness and a Deacon watch for stuck workers | Workers; the Refinery merges | The merge queue's tests | +| **Cursor long-running agents** (P) | None during a run | A harness that runs planners, workers and a judge in cycles | A shared branch | Planners create tasks; workers "grind on their assigned task until it's done, then push"; a judge decides whether to continue | Workers push to one branch | The judge, and the project's own tests | +| **Anthropic parallel-agent C compiler** (S for mechanism) | None | None: a loop restarts each agent | Lock files in a directory of the repo | Agents claim a task by writing a lock file and sync through git | Each agent | A large test suite, and a reference compiler used as an oracle | +| **Kubernetes Prow and Tide** (P: Prow docs; not agentic) | Humans approve with commands that set labels | Always-on services; Tide syncs with GitHub every minute | Labels and GitHub's own state, read by search queries | Plugins set labels; Tide merges whatever the queries return | Tide | Required tests re-run against the latest base, in batches | + +Research frameworks sit to one side of these. MetaGPT passes structured documents between roles in +a fixed order through a shared message pool; ChatDev and AutoGen have roles converse. None is +presented as a production system. + +### 13.2 How roles interact, across all of them + +1. **Conversation.** Roles talk in natural language (AutoGen, ChatDev). No production-leaning system + above uses it. +2. **Structured handoff.** Roles pass typed documents in a declared order (MetaGPT, and the BMAD + method this repo vendors). ADR-0013's Task and Result are this pattern. +3. **A shared work ledger.** Roles never address each other. They take work from a durable store and + put results back (Beads, Symphony's tracker, the compiler's lock directory, Tide's labels). Every + system above that runs unattended for long uses this. ADR-0013's State record and Ledger are this + pattern. +4. **Hierarchy.** One role plans, others execute, another judges. Cursor reports that equal peers + became risk-averse: "No agent took responsibility for hard problems or end-to-end + implementation." +5. **Supervision as a role.** Gas Town makes its watchdogs first-class. Symphony's tick detects a + stalled run after five minutes without an event and retries with backoff. + +The POC uses a sixth pattern that none of these chose: roles signal each other by side effect, with +comments, labels and markers on a surface shared with humans and third-party bots (§3.2). + +### 13.3 Where the proposal is stronger than the comparables + +- **Effects are applied outside the model.** In Symphony, Cursor's harness and the compiler + experiment the agent itself writes. Symphony's specification leaves the trust boundary to each + implementation and its README is reported to describe it as a preview for trusted environments. + ADR-0013's credential boundary per job is stricter than any comparable, and it is the right + strictness for public repos that take text from strangers. +- **Review independence is enforced by the platform.** The comparables either remove review + (StrongDM), or let agents review agents with no separation of identity (OpenAI). ADR-0013 keeps a + gate that the author's identity cannot satisfy. +- **State survives a restart with its history.** Symphony deliberately keeps scheduler state in + memory and re-derives it. That is simple, and it means attempts, spend and dispositions are not + recorded anywhere. ADR-0013's Ledger makes cost per outcome and "why is this stuck" answerable. +- **It releases itself safely.** None of the comparables describes versioning, staged rollout or + rollback of the factory. They are single-team tools. This repo serves twelve. +- **It is frugal.** StrongDM's stated benchmark is $1,000 a day in tokens per engineer. This fleet's + measured spend is $196–280 a week (§2). + +### 13.4 Where the proposal is weaker than the comparables + +- **Every comparable that orchestrates has an always-on orchestrator.** Symphony ticks every 30 + seconds. Tide syncs every minute. Gas Town runs as a standing process. Each holds its claim set in + one process, so double dispatch is avoided by a variable, and a timer is a timer. ADR-0013 + emulates that process with wake-ups, concurrency lanes and a cron that drops about 60% of its + ticks. Symphony's own non-goals include being a "general-purpose workflow engine or distributed + job scheduler"; ADR-0013 builds one out of Actions. +- **Files in git as the work ledger is a design another project abandoned.** Beads kept a local + database synchronized through a JSONL file in git. Its maintainers record stale overwrites, + resurrected deletions and split-brain failures from that period, and moved to a database with + merge semantics. ADR-0013's store differs in one way that matters: only the Controller writes it, + so there is no multi-writer merge. Concurrent Controller runs still race on one ref. +- **Assurance rests on review by the same model family that wrote the change.** StrongDM's answer + to agents that "return true" or rewrite a test is validation the author cannot see. The compiler + experiment needed a reference oracle. ADR-0013 verifies that a claimed commit exists. It adds + nothing that tells a correct change from a plausible one, and this repo's holdout sets are public. +- **A mandatory quality gate is a bottleneck by construction.** Cursor built an integrator role for + quality control and removed it: it "created more bottlenecks than it solved." The contexts differ, + since Cursor was building greenfield on one branch with no protected trunk. The observation still + applies to a single approval that five gates can withhold. +- **It is much larger.** Symphony is one specification. The compiler harness is a shell loop. + ADR-0013 requires a kernel, a harness, a controller and eight further ADRs, for one maintainer. +- **Nobody has shown this problem solved.** A September 2026 synthesis of industrial evidence + concludes that no company has implemented a full control plane of this kind, and cites agents + raising commits by 180% while releases rose 30%. The comparables are single-repo, mostly + greenfield and run inside one trusted team. This factory is multi-repo, brownfield and public. + +### 13.5 Choices the proposal makes that others abandoned + +| Choice in ADR-0013 | Who abandoned it | Why it may still hold here | What would show it does not | +|---|---|---|---| +| Work state as files in git | Beads | Single writer identity; no merge of concurrent edits to one record | Rejected pushes per decision, or time to clone the store, rising with load | +| Leases taken by optimistic write | Cursor moved past it | Cursor called optimistic control "simpler and more robust" than locks; its remaining problems were about who owns hard work, which hierarchy answers | Re-decisions per admitted Run above a small constant | +| A quality gate every change must pass | Cursor | The ruleset requires an approval, and the repos are public | Share of lead time spent waiting at the gate | +| Agents reviewing agents as the main assurance | StrongDM | No budget for a scenario estate, and no private place to hold one | Escaped defects per hundred merges (SM-C1) rising with the autonomy rate | +| No standing process | None of them tried it | Constraint C-4 | The reopen triggers in §7.5 | + +### 13.6 Sources + +- StrongDM Software Factory: ; Simon Willison's account, + +- OpenAI Symphony specification: ; OpenAI, + "Harness engineering", +- Gas Town: ; + Beads storage: +- Cursor, "Scaling long-running autonomous coding": +- Anthropic, "Building a C compiler with a team of parallel Claudes": + +- Prow and Tide: +- Cemri et al., "Why Do Multi-Agent LLM Systems Fail?": +- Bhati, "Beyond Code Generation": +- MetaGPT: ; AutoGen: + +--- + +## 14. Adversarial review: the ceiling, and where it fails first + +An independent pass attacked ADR-0013 after §§4–13 were written. It had no part in the design. Its +platform claims were read from GitHub's documentation on 2026-10-10 and its repo claims re-checked +here. Figures marked *est.* are estimates with the stated range. + +**Verdict: approve with named changes, not as written.** Four decisions are sound and do not depend +on the substrate: the run contract and credential split, agent identity, content-bound approval, +and verification that binds. Two were a bet taken before the measurements that would justify it: +the git-backed store, and a Controller woken per event on Actions. The first draft also carried no +budget and no stopping rule. §14.7 lists what was changed in the ADR as a result. + +### 14.1 The ceiling + +| Rank | Constraint | Where it binds | What ADR-0013 does to it | +|---:|---|---|---| +| 1 | **The maintainer's attention** | About 30–36 hosting-repo merges a week, the observed plateau (*est.* from a time model; see below) | Nothing directly. It rises only if approvals issue *and* can be trusted | +| 2 | **First-pass correctness** | The autonomy rate at which escaped defects exceed what one person can absorb (§14.4) | Nothing. Review depth is out of scope | +| 3 | **The LLM weekly window** | About 100–150 org merges a week at POC efficiency (*est.*, ±50%); reached or approached in about two of the last eight weeks | Cuts waste under it. Adds new claims on it: the rewrite itself, and an eval pass per promotion | +| 4 | **20 hosted-runner jobs, org-wide** | Saturated at peaks today; about 55–80 hosting-repo merges a week sustained (*est.*) | 2–3× from stopping the rebase fan-out and sleeping runners, less a 10–25% tax for wake handling | +| 5 | **App API quota**, 5,000 an hour | Holds only if an evaluation costs at most about 3 GraphQL points; 3.4× over at the POC's call style at peak | A separate quota is a real gain. The margin depends on a read budget nobody has written | +| 6 | **Git as the ledger** | 8–18 pushes a minute at peak against GitHub's recommended maximum of 6 per repository | A new constraint the POC does not have | +| 7 | The merge queue | Above 1,000 merges a week | Does not bind | + +**The architecture removes waste under constraints 3 to 6. It does not raise constraints 1 or 2, +and those are the ones that bind.** + +The reasoning behind rank 1: merges in the hosting repo have been flat at 14–36 a week since late +August while workflow runs rose about fivefold (§2). A ceiling that does not respond to five times +more machine activity is not a machine ceiling. In the latest week 21 of 36 merges went in by hand +without a standing approval, and holds were cleared by the owner account 64 times across 150 merged +pull requests. A model of that work (an audit per bypass merge, hold triage, promotions, filing and +steering) comes to 10–12 hours a week at about 36 merges. The model is an estimate, and the shared +login means nothing measures it. One week after agents move to their own identity it can be counted. + +The runner arithmetic, because it is the least obvious. The free plan allows 20 concurrent jobs. +One push to a pull request in the hosting repo starts about 56 (counted from +`.github/workflows/*.yml`; 66 across all pull-request event types). Auto-rebase updates every behind +pull request by default (`.github/workflows/auto-rebase.yml:25-30`), so with 14 open, one merge to +`main` starts 784 jobs: about 28 minutes of total saturation (*est.*, range 13–42). About a fifth of +sampled job time is a runner asleep, waiting for a rate-limit reset +(`.github/workflows/pr-review-delayed-retry.yml:16`, `:79`). + +What the design adds to that: a comment that needs no work costs one runner job today and two +skipped listeners. Under ADR-0013 as first drafted it costs three runner jobs: the consumer's +ingress, ring resolution in the entry shim, and the decision. Scaled from the measured event mix, +that is at least 12,500 wakes a week from the hosting repo alone. Unfiltered, wake handling would +take about 18% of the org's runner slots on average and about 83% at the measured peak. With events +authored by the factory's own identities dropped at the ingress, 7% and 33%. The filter matters +because the fleet is its own largest event source: 85% of sampled comment, review and push events +were authored by the two agent accounts, and under ADR-0013 those become App-authored events that +start workflows. + +### 14.2 Where it fails first + +Ranked by how early and how surely each appears. + +1. **The migration starves.** The ADR freezes the POC to defect fixes. Bug-labelled issues arrive at + 21–36 a week and 31% of defect issues name a prior one, so holding the POC still needs about + 30–52 fix pull requests a week. The line delivers 29–36. The rewrite is on the order of 100–150 + pull requests, each needing the approval that currently does not issue. The repo's record on + finishing migrations is poor: ADR-0007 is accepted and not deployed here, ADR-0009 serves one + surface twice, `release/registry.yml` is an orphan, the budget breaker was closed and cannot fire + (`petry-projects/.github#1155 — The token-budget breaker is unreachable`), and five consumers + still pin a legacy tag (`petry-projects/.github#1125 — Fleet pr-review callers pin legacy bare + pr-review/stable and are unmanaged`). *Signal:* the weekly share of merged pull requests that + touch `factory/`; under 30% for three weeks means the freeze is nominal. *When:* the first day + of step 1. +2. **Step 1 is not harmless.** Observing by per-event wake competes with the line for the same 20 + slots (§14.1). A status block rewritten on every evaluation can reach GitHub's limit of about 500 + content-creating requests an hour, which would throttle every factory write. The ledger exceeds + the recommended push rate at peak. A status block that prints a "last evaluated" time is also a + self-wake loop if the ingress hears comment edits. *Cheapest mitigation:* run step 1 as a + **poller**: one scheduled job that evaluates every open item in one process and writes one commit + per pass. An observer does not care about dropped ticks. +3. **Approvals still do not issue after the review cutover.** See §14.3. *Kill criterion:* if the + share of agent merges with a standing approval on the first cut-over repo is under 70% after two + weeks, stop the fan-out. +4. **Effects are not idempotent, and no brake sat outside the Kernel.** GitHub writes carry no + idempotency key. If `apply` dies after applying some Effects and before the ledger commit, the + next evaluation either re-applies (duplicate comments and reviews) or re-admits (a second model + Run and duplicates). The usual cure is a marker in the body, which the first draft forbade + reading. Separately, the first draft put Pause "in the admission function and nowhere else", so a + broken Kernel would be the thing evaluating Pause, where today the pause is a job condition the + platform evaluates (`.github/workflows/dev-lead-reusable.yml:116`). With App-authored events + starting workflows, a Kernel defect is an org-wide loop: one Run per five minutes on 14 items is + 168 an hour, and a week's window of about 600 Runs is gone in under four hours. The precedent ran + four days unnoticed (`docs/postmortems/2026-06-pr-860-runaway.md`). +5. **The credential boundary stops at the hub.** `apply` pushes model-authored commits to a branch + in the consumer. GitHub then runs that branch's own `pull_request` workflows with the repo's + secrets. No workflow here uses an `environment:`, and nine pass `secrets: inherit`. A + model-authored edit to a workflow, or to code a workflow runs, can read whatever that repo can. + This is not new exposure, since today the model holds the owner credential outright. It is a + boundary the first draft claimed and did not have. Related: the credential a consumer uses to + send a wake needs `Contents: write` or `Actions: write` on the hub, so it is not dispatch-only; + and `verify` runs consumer code in the hub's context, where a `repository_dispatch` run may write + default-branch caches that every later job can restore. +6. **"CI finished" is hard for one frozen ingress file to hear.** `check_run` and `check_suite` do + not start workflows for suites created by Actions, and `workflow_run` must name the workflows it + watches (`.github/workflows/pr-auto-review.yml:32-34` names `"CI"`). A byte-identical consumer + file works only if every repo's CI carries a standard name. Otherwise the commonest transition, + checks green to review, waits for a sweep that fires every 3–8 hours in practice + (`petry-projects/.github-private#2170 — Retry and sweep crons run every 4–8 hours instead of + every 2 h / 15 min`). +7. **The identity cutover meets an untested rule.** The live ruleset sets + `require_extra_approval_for_unattributed_changes: true`. GitHub documents it for agent-authored + pull requests not attributed to a person. If it applies to a pull request authored by the factory + App, every agent change needs two approvals and the org has one non-human approver. One test pull + request settles it. +8. **Promotion needs the scarcest resource.** In the weeks the model window is spent, no Release can + be promoted, so a bad `stable` cannot be fixed forward. Rollback must run nothing from the Release + it is rolling back. +9. **The model CLI changes under every Role at once** (§12, item 11). + +### 14.3 Is the largest failure class caused by architecture? + +Mostly not, and this is the finding that most changes the picture. + +- **The same implicit control plane once delivered.** The week of 2026-07-12 saw 79 merges at a + median of 0.9 hours; the week of 08-02, 27 at 0.9 hours. What changed afterwards was the number of + things an approval must survive. Class A went from 1 to 17 to 45 issues over three four-week + windows as the maintainer-comment gate was added and then redesigned so that every pull-request + comment, "from human maintainers and bots alike", needs a verified disposition + (`scripts/lib/maintainer-comment-gate.sh:15-30`). +- **Those gates had reasons**: zero-diff merges, trunk corruption, a runaway. The trade is + throughput against assurance, and it does not depend on where state lives. +- **The environment does not change.** GitHub still dismisses a stale approval and still bars the + last pusher from approving. Nine third-party reviewers still post free text, edit it in place and + re-post on every commit: 1,219 review events from four of them on 174 pull requests in one week. +- **"One gate function over facts" still needs something to turn free text into facts.** A finding + from a review bot is a comment body. A notice is recognised today by a per-bot pattern + (`scripts/lib/reviewer-sources.tsv`). The first draft of the ADR both required findings to be + classified and forbade code from reading a comment body. The architecture gives the heuristic one + home. It does not remove it. + +What the architecture does fix in this class: a Role dismissing its own approval, five places that +can withdraw one, and re-review of unchanged content. Those are real. They are a minority of the 45. + +**One policy change would likely do more for this class than the rewrite: let human comments and +review threads gate an approval, and treat third-party reviewer output as input to the one review +pass.** It can be tried in weeks. + +### 14.4 The quality ceiling + +Author and reviewer are the same model family. `verify` runs the consumer's own tests, which the +author can edit. A held-out eval set in a public repo is held out by convention. The design verifies +that a claimed commit exists. It does not verify that the change is right. + +Evidence that it often is not: ten zero-diff pull requests merged and closed their issues +(`petry-projects/.github-private#1620 — dev-lead merges zero-diff PRs that close their issues`); +tests were edited to match the code (`petry-projects/.github-private#2013 — dev-lead fix-reviews: +posts "Fixed" replies for changes that were never pushed, and edits tests to match its own code`); +23% of merged pull requests took ten or more commits. + +Let *d* be the share of merges that carry a defect, *a* the share merged without a human audit, and +*c* the share of defects an audit catches. Escaped defects per week are *M · d · (a + (1 − a)(1 − +c))*. With 36 merges, *c* = 0.7 and *d* = 0.3: + +- today, with *a* about 0.4: about 6 a week; +- at the PRD's 85% autonomy: about 10 a week; +- the PRD's defect target is 5. That needs *a* ≤ 0.23, or, at 85% autonomy, *d* ≤ 0.155. + +*d* is the weak number. It is inferred from bug-issue inflow, which also counts process gaps found +by audit, so 0.3 is probably high; the plausible range is 0.15–0.4. The conclusion holds across the +range: **the PRD's autonomy and defect targets are a pair, and on current first-pass correctness +they cannot both be met.** The autonomy rate consistent with the defect target is nearer 25–40% than +85% until correctness improves. StrongDM reached the same wall and moved acceptance to scenarios the +author cannot see (§13). + +### 14.5 The alternatives at their strongest + +**Configuration, identity and gate scope, then re-measure.** Step 0 alone removes the largest +measured waste: 784 jobs and about 28 minutes of runner saturation per merge. The App identity alone +gives a separate API quota, lets the maintainer review agent pull requests, and makes the existing +loop budget count dev-lead with no code change, because the budget already treats a login ending in +`[bot]` as automation (`scripts/lib/pr-automation-budget.sh:54-59`). The gate-scope change of §14.3 +addresses the largest failure class. Together they cost weeks. Against this: lost events, the +credential in the model, two release regimes and no ledger all remain, and this repo's local fixes +have a record of moving failures. *What decides it:* four weeks after those three changes, read the +share of agent merges with a standing approval, the inflow of classes A and D, runs per merge and +median lead time. If approvals stand on 70% or more and the median is under 8 hours, the case for +the rest of the rewrite rests on classes B, C and D alone and should be re-argued on those. + +**A poller in place of per-event wakes.** Symphony, Tide and Zuul all poll or hold state in one +standing process, and in all three the control state is disposable: it is rebuilt from the platform +on restart. ADR-0013's first draft chose two things none of them did: no standing process, and a +second authoritative store that cannot be rebuilt. A poller needs no inbound endpoint, no file in +any consumer, no credential in any consumer, and no wake handling. Polling twelve repos once a +minute is 60–300 GraphQL points an hour, under 6% of the App's limit. It can run in three places: + +- as a scheduled workflow in the hub. Nothing new to operate. Scheduled ticks are dropped about 60% + of the time, so a five-minute schedule gives an effective period nearer fifteen; +- as one long-lived job in the hub that ticks in a loop and restarts itself. This gives a + thirty-second tick and an in-memory claim set for one of twenty runner slots. GitHub's terms bar + using Actions "as part of a serverless application" and a burden "disproportionate to the + benefits", and do not address long-running jobs. Whether this is acceptable use is an open + question, and it should not be relied on until that is answered; +- on a small host. This is the option constraint C-4 excludes. + +*What decides it:* the Kernel does no I/O, so step 1 can run it under the wake path and under a +poller at once and compare latency, runner minutes, API points per evaluation and push retries. + +### 14.6 What did not bite + +- **The merge queue's rate:** under 3% used. +- **Ledger size over 180 days:** 0.4–1.5 GB against a recommended 10 GB. The shape is the problem: + one file per event passes the recommended 3,000 entries per directory within days unless sharded, + and "append-only" conflicts with "retention" unless the store is partitioned by time. +- **Two Runs on one item after a lease expires:** the platform lane prevents it. The cost is a + blocked org-wide slot until the lease times out. +- **Giving up the default token's loop brake:** the POC's main writer never had it. The identity + change improves loop counting. +- **Actions trigger limits:** 1,500 events and 500 queued runs per 10 seconds per repo are orders of + magnitude away. +- **Python with the standard library only:** no objection found. + +### 14.7 What changed in ADR-0013 as a result + +| # | Required change | In the ADR | +|---|---|---| +| R1 | The git-backed store and per-event wakes are provisional, with thresholds written before step 1; the same Kernel also runs under a poller | Decision 11 | +| R2 | The ingress drops factory-authored events, holds a per-item concurrency group and subscribes to nothing that runs a branch copy of itself; the status block is throttled | Decisions 3 and 6 | +| R3 | Brakes outside the Kernel: a pause variable on the entry shim, a Run ceiling from a count the Kernel does not write, the loop count from GitHub's record of actors | Decision 3 | +| R4 | Effect application is specified: recovery keys, an intent record before a non-idempotent Effect, lease release under `always()`, rollback that runs nothing from the bad Release | Decisions 4 and 6 | +| R5 | The comment-body boundary is restated so it can hold; resolving a thread is a human transition. The first response made third-party output input only; the maintainer then directed that reviewers be integrated through a contract, so structured Findings gate and comment bodies do not (§15.5) | Decisions 2, 7 and 12 | +| R6 | Environment-scoped secrets; `run` and `verify` without permissions or cache writes; `apply` refuses changes to workflows, code owners and declared test commands without a human transition | Decisions 4 and 5 | +| R7 | The migration has a dated limit, a POC defect policy and a kill criterion, set by the maintainer before step 1 | Decision 10 | +| R8 | Escaped defects are baselined before any step raises autonomy; evals exist for the approving and code-writing Intents before their cutovers | Decisions 6 and 10; PRD NFR-3 with NFR-10, FR-59 | +| R9 | Reopen triggers for the model CLI's default, a native orchestrator from the platform vendor, and plan changes; the off-Actions trigger is evaluable from step 1 | Consequences | +| R10 | Three assumptions are settled in step 0 on a canary repo | Decision 10 | + +### 14.8 Assumptions the adversarial pass could not close + +- That the 20-job limit is shared by all twelve repos. The documentation does not say; a queue of + more than 200 runs on 2026-10-10 is consistent with it. +- Whether a `repository_dispatch` call counts against the content-creation limit. +- Whether a deployment-branch restriction on an environment blocks secret access from other + branches. +- Event rates in consumer repos, which were scaled from their share of merges. +- Job-seconds per push, and the defect rate per merge. + +--- + +## 15. The convergence hypothesis: a council of Lenses + +The maintainer's core hypothesis is that it takes a team to reach an optimal outcome: a council of +models, tools and perspectives working across research, analysis, architecture, design, +implementation, validation, release and operation. This section asks two questions. Did the first +draft of ADR-0013 consider it? And what does the evidence say? + +### 15.1 Did the proposal consider it? + +No. ADR-0013 as first drafted is about the control plane. It made every Role data, gave every Run +one contract and put one gate function in the Kernel. It listed review depth as not addressed, and +the PRD listed it as a non-goal. The adversarial pass then named first-pass correctness as the +second ceiling the proposal does not raise (§14.1). A council is the obvious lever on that ceiling, +and nothing in the first draft evaluated one. + +The POC, by contrast, already runs a large council: + +- the review cascade: triage, then a deep review with four specialist prompts (`prompts/deep-review-logic.md`, + `-performance.md`, `-security.md`, `-style.md`) in parallel with a second opinion from another + vendor's model (`prompts/rubber-duck.md`; `docs/engine-configuration.md:15`), then synthesis and + an audit; +- nine advisory personas (`personas/`), each a different professional perspective; +- nine third-party reviewers (`scripts/lib/reviewer-sources.tsv`); +- CI tools that are Lenses in all but name: CodeQL, SonarCloud, the test suites. + +So the hypothesis has been running in production for months. It has not been tested, because the +council interacts through discussion and nothing measures what each member contributes. The +reviewer scorecard's own-automation section reads zero and its miss rate reads 0 of 0 (§2.2). +What can be measured is the cost of the protocol: 343 review dismissals on 150 merged pull +requests, commits per merged pull request up from 2–4 to 7–10, and the largest failure class in the +issue history (§2.3, class A). + +ADR-0013 now carries decision 12, and the PRD carries §6.11, as a result of this section. + +### 15.2 What the evidence says + +Each source is marked **P** (read directly) or **S** (secondary summary). Preprints are named as +such. + +#### In favour of a council + +| Finding | Evidence | +|---|---| +| An independent evaluator beats self-evaluation | Models rate their own output above equal-quality output from others, and the effect grows with their ability to recognise it (Panickssery et al., NeurIPS 2024, S). In a controlled study, one model's solo pass rate of 71.6% rose to 84.5% with self-review and to 89.7% with review by a stronger model (Xiang et al., preprint, P). Anthropic's long-running harness separates a generator from an evaluator for this reason (S) | +| Different Lenses find different defects | A second model's findings overlapped a same-model reviewer's by 37–46%, against 52–62% between two runs of the same model. One same-model plus one cross-model review matched 56.7% of planted errors, against 42.7% for two same-model reviews (Song, single-author preprint, P) | +| Parallel Lenses help on work that divides | Centralised multi-agent coordination improved a parallelisable analysis task by 80.8% (Kim et al., Google and MIT, P). Anthropic reports a multi-agent research system beating a single agent by about 90% on research tasks, at roughly fifteen times the tokens (S) | +| Structured handoff beats conversation | MetaGPT's authors report roughly half the tokens per line and higher executability than a chat-based framework, and attribute it to passing documents instead of dialogue (S; the authors' own comparison) | + +#### Against a naive council + +| Finding | Evidence | +|---|---| +| Collaboration hurts work that is one interdependent piece | Every multi-agent variant degraded a sequential task by 39–70% (Kim et al., P). Coding agents given two features in one codebase succeeded about 30% less often cooperating than working alone; the authors call it the curse of coordination (CooperBench, ICLR 2026, S) | +| A weaker Lens makes a stronger author worse | The stronger model alone passed 91.4%. Reviewed by the weaker model it fell to 82.8%, with an 11.2% regression rate. Reviewing itself changed nothing (Xiang et al., P) | +| Returns fall as the single agent gets better | Coordination shows diminishing or negative returns once a single agent's baseline passes about 45% (Kim et al., P) | +| Mixing models is not the same as diversity that helps | Using one strong model several times beat mixing different models by 6.6% on one benchmark, because adding models lowered average quality (Li et al., "Rethinking Mixture-of-Agents", S) | +| Models share blind spots | When two models are both wrong they give the same wrong answer about 60% of the time, and stronger models correlate more, across vendors (Kim et al., ICML 2025, S) | +| Discussion adds little over independent votes | Majority voting accounts for most of the gain credited to multi-agent debate; debate alone does not improve expected correctness (Choi et al., NeurIPS 2025, S) | +| Councils fail in their protocol | A study of more than 1,600 traces attributes multi-agent failure mainly to system design and inter-agent misalignment, not to weak models (Cemri et al., S) | + +Two gaps in the literature matter here. None of these studies compares a model Lens with a Lens +grounded in execution, such as a test run or a static analyser; the claim in §15.3 that such Lenses +are the most independent is an inference from the correlated-error result and from what StrongDM +and the compiler experiment had to do (§13). And almost all of it is measured on benchmark tasks, +not on a brownfield repo over months. + +### 15.3 Verdict on the hypothesis + +Neither proven nor disproven as stated. It is conditionally supported, and the conditions are +specific. + +**Supported:** an outcome judged by independent Lenses is better than one judged by its author; a +council raises coverage when its members can see different things; and a council is the right shape +for work that divides, which is most of research, analysis and review. + +**Not supported:** that more members are better; that members improve an outcome by talking to each +other; that a council should write the change together; or that a different model brand is, by +itself, an independent view. + +Put together, the form the evidence supports is: **one author, surrounded by independent Lenses +that report in parallel through a contract to something that decides.** That is a council. It is +not a conversation. + +Three conditions carry most of the weight: + +1. **Independence has to be real.** Different information, different tools or a different stage of + the work make a Lens independent. A second model reading the same diff with the same prompt + mostly does not. +2. **A Lens must be at least as capable as what it judges, or grounded in something the author is + not.** Otherwise it subtracts. +3. **The protocol decides the result.** Structured Findings and a deterministic aggregate converge. + Free-text discussion among agents does not, and this repo's own telemetry is the evidence. + +The third condition is the maintainer's first point in other words: integrate through a contract, +render for humans, interpret structured data. + +This review is itself a small instance. The options analysis was written by one author (§§4–11). +An independent pass with a fresh context and an adversarial brief then judged it and required ten +changes (§14.7). A comparison against outside systems added a third view (§13). None of the three +conversed; each reported in a document and the changes were aggregated afterwards. One case proves +nothing, and it matches the pattern. + +### 15.4 A starting shape for the council, by stage + +This is where the evidence points, not a decision. ADR-0013 leaves composition to a follow-on record +decided from measurement. + +| Stage | Shape the evidence supports | Strength of evidence | +|---|---|---| +| Research and analysis | Lenses in parallel, one synthesiser | Strong | +| Architecture and design | One author; an independent adversarial critic; a comparison against outside practice | Moderate | +| Planning | One author; a critic checking the plan against the requirements | Moderate | +| Implementation | One author, no co-writers | Strong, against collaboration | +| Validation | Several Lenses that differ in kind: the repo's tests, static analysis, an acceptance signal the author cannot edit, a model reviewer | Moderate; the grounded-Lens part is inference | +| Review | A model reviewer at least as capable as the author; a second model only if it measurably adds Findings | Mixed | +| Release and operation | Deterministic gates and monitors as Lenses; a model Lens for diagnosis | Practice, not research | + +### 15.5 Third-party reviewers under a contract + +The reviewer registry already points this way. It records, per reviewer, whether it creates review +threads and the name of the check run it reports through, and the advisory gate already counts a +completed check run as a response (`scripts/lib/reviewer-sources.tsv`, columns `creates_threads` +and `check_run_name`). What it also carries is a body pattern per reviewer for recognising notices, +which is the discussion channel this decision retires. + +GitHub offers three structured surfaces a reviewer can report through: + +| Surface | What is structured | Can it gate | +|---|---|---| +| A check run with annotations | Conclusion; per annotation a path, lines, a level and a message. Only a GitHub App can create one | Yes, by conclusion and by annotation | +| Code-scanning alerts from a SARIF upload | Rule, severity, location, a stable alert identity and state | Yes | +| A review thread | An identity, a path and line, resolved and outdated flags. The body is prose | Yes, by the thread's state. Severity is not structured | +| A summary comment | Nothing | No | + +Which of the nine current reviewers expose which surface is established in §16. The rule in +ADR-0013 does not depend on the answer: a reviewer is as influential as its contract surface +allows. + +### 15.6 Sources + +- Kim et al., "Towards a Science of Scaling Agent Systems": +- Xiang et al., "Cross-Model LLM Code Review" (preprint): +- Song, "When Does a Second Model Help?" (preprint): +- "CooperBench: Why Coding Agents Cannot be Your Teammates Yet" (Stanford and SAP Labs): + +- Li et al., "Rethinking Mixture-of-Agents": +- Choi et al., "Debate or Vote": +- Kim et al., "Correlated Errors in Large Language Models": +- Panickssery et al., "LLM Evaluators Recognize and Favor Their Own Generations": + +- Cemri et al., "Why Do Multi-Agent LLM Systems Fail?": +- Anthropic, "How we built our multi-agent research system": + ; "Harness design for + long-running application development": + +- MetaGPT: +- GitHub, check runs: ; SARIF upload: + + +--- + +## 16. Reviewer surfaces and a starting council + +§15.5 left open which reviewers expose a structured surface. This section answers it for the nine +installed reviewers from two directions, what each emits on this org's pull requests and what its +vendor documents, then evaluates a wider candidate pool, including SonarQube and Gitar, and +proposes a council to start measuring from. + +### 16.1 Method and limits + +- **Observed surfaces:** 16 recent pull requests across four repos (hosting repo, standards repo, + TalkTerm, broodly), read once through GraphQL on 2026-10-10: every check run with its app, + conclusion and annotations; every commit status; every review with its state; every review thread; + every pull-request comment. +- **Observed contribution:** 540 review threads on 24 pull requests, with the replies, classified by + the author agent's disposition marker. Volume and refusals come from the org's own weekly reviewer + scorecard of 2026-10-05 (174 eligible pull requests). +- **Documented surfaces:** each vendor's own documentation, read on 2026-10-10 through a fetcher + that returns a model-extracted rendering. Names and thresholds should be re-checked against a real + pull request before code depends on them. +- **Limits.** The samples favour busy pull requests. "Acted on" means the author agent claimed a fix; + §14.4 is the evidence that this is not the same as the finding being right. Thread lists were + capped at 60 per pull request. No sample measures a reviewer's false-negative rate. + +### 16.2 The nine: what each emits here, and what its vendor documents + +| Reviewer | Check run or status seen here | Annotations | Review state | Threads | Severity | Findings outside GitHub | Summary comment | +|---|---|---|---|---|---|---|---| +| **SonarQube Cloud** | Check `SonarCloud Code Analysis`; conclusion follows the quality gate | **Yes**, per line, with a level | None | None of its own | In its API | **Yes**: issue key per finding; Web API and MCP. Its findings also arrive as **code-scanning alerts** here | Yes, re-posted on every push | +| **cubic** | Check `cubic · AI code reviewer`; neutral seen on two runs; meaning undocumented | None | Comment only | Yes; resolves its own when fixed | `P1`–`P3` as text, plus a confidence value in a hidden comment | **Yes**: MCP `get_pr_issues` with issue ids, `update_pr_issue_status` | Generates the description | +| **CodeRabbit** | Commit status `CodeRabbit`; documented as "review ran", not "findings exist" | None | **Approve and request-changes seen** (24 and 31 in one week) | Yes; resolves its own when fixed | Category and severity as emoji text | Partial: CLI prints thread roots as JSON; REST is Enterprise | Yes, edited in place | +| **Graphite** | Check `Graphite / AI Reviews`; not documented by the vendor | None | Comment only | Yes | None | None | None seen | +| **Gemini Code Assist** | None | None | Comment only | Yes | A priority badge image | None documented | None seen | +| **Codex** | None | None | Comment only | Yes | A `P1`/`P2` badge image | None | Usage-limit notices | +| **CodeAnt** | None seen; documented only for its scanner gates | None seen | Comment only seen; an approving review is documented as an option, off by default | Yes | Severity and category as text | Its CLI returns its own comments and reviews as JSON, with a filter on resolved; no severity field is documented | Yes, a risk summary | +| **Copilot code review** | None | None | None seen | None seen | A text label, per its docs | None | None | +| **Qodo** | None | None | None seen | None seen | Text labels, per its docs | MCP `findings`, fields undocumented | Only "trial has ended" notices | + +For comparison, **CodeQL** has every surface: a check whose conclusion reflects findings, +annotations, alerts with a stable number, a state and a rule severity, and a REST API. + +What this means for the contract in ADR-0013, decision 12: + +1. **SonarQube Cloud has the strongest contract of the nine, and the POC reads its weakest + surface.** The registry classifies it as "top-level quality-gate comment only" and recognises a + clean result by a pattern over the comment body (`scripts/lib/reviewer-sources.tsv`). The same + information is in a check conclusion, in annotations, and in code-scanning alerts that carry a + Sonar issue key. 34 of the hosting repo's last 100 code-scanning analyses came from SonarCloud. + The vendor documents that sync as an Enterprise feature; it is running here on a public repo. +2. **No LLM reviewer publishes findings through SARIF, and every one carries severity as text in the + comment body.** A contract that does not parse bodies cannot read severity from a thread. For + thread-surface reviewers the gate is therefore the thread's own state: open or resolved. Severity + is available only from a reviewer with a findings API, which among the nine means cubic. +3. **Only one LLM reviewer emits a structured verdict.** CodeRabbit submits approving and + change-requesting reviews. The rest submit comment reviews, which say nothing a gate can read. +4. **A green check is not evidence of a review.** CodeRabbit documents a *passing* check named + "Review rate limited" when it is throttled, and its main status tracks progress. Graphite's and + cubic's checks exist and their meaning is not documented. +5. **No reviewer here is configured as code.** There is no `.coderabbit.yaml`, `cubic.yaml`, + `.codeant/` or `.pr_agent.toml` in the repo. Each runs on settings held in its vendor's console. +6. **CodeAnt has more of a contract than the first reading found.** Its documentation was re-read + directly on 2026-10-10, without the summarising fetcher. Three things are documented and not in + use here. An "Auto Approve PR" setting submits an approving review once every thread CodeAnt + opened is resolved, pinned to the commit it reviewed. A status check fails a pull request on + scanner quality gates, which are metrics and not review findings. A CLI lists CodeAnt's own + comments and reviews as JSON and filters on resolved. What it still lacks is severity as a typed + field, annotations, and code-scanning alerts. On `petry-projects/.github-private#2225 — docs: + factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)` it emitted two + comment reviews, three threads and two comments, and no check. An approving review from a + third-party App should be tried on a canary repo before it is enabled, to see whether the + platform counts it toward the required approval. + +### 16.3 What each contributed + +Scorecard columns are from the week to 2026-10-05. Thread columns are from the 540-thread sample. +"Shared" is the share of a reviewer's flagged locations that another reviewer also flagged within +about six lines. + +| Reviewer | Reviews | Refused | No response (of 174) | Threads sampled | Acted on | Shared | Severity mix | Model family | Free for public repos | +|---|---:|---:|---:|---:|---:|---:|---|---|---| +| cubic | 222 | 105 | 67 | 173 | **71%** | 32% | P1 32, P2 100, P3 41 | OpenAI and Anthropic | Yes, fair-use allowance of unstated size | +| Codex | 74 | **249** | 33 | 142 | 64% | 25% | P1 64, P2 78 | OpenAI | No; needs a paid ChatGPT plan | +| CodeRabbit | 401 | 155 | **3** | 67 | 59% | 33% | Major 13, Minor 22, Trivial 3 | Not disclosed | Yes; 1–10 reviews an hour per repo | +| Gemini Code Assist | 118 | 15 | 49 | 60 | 60% | **16%** | high 12, medium 26, low 22 | Google | Free while in preview, through Google Cloud | +| CodeAnt | **453** | 21 | 64 | 53 | 50% | **43%** | Major 48, Critical 4 | Not disclosed | No free tier found in its documentation; free for this org, per the maintainer | +| Graphite | 22 | 8 | 101 | 15 | 66% | 26% | none | Anthropic and OpenAI | No, for an org's repos | +| SonarQube Cloud | 116 | 0 | 59 | 8 alert threads | 0 of 8 | — | by rule | None; rule-based | Yes | +| Copilot | **2** | 0 | 172 | 0 | — | — | — | Not disclosed | No | +| Qodo | 143 notices | 0 | 31 | 0 | — | — | — | Not disclosed | No, below 200 stars | + +Readings, with the caution of §16.1: + +- **Volume and value diverge.** CodeAnt submitted the most reviews, had the lowest share acted on, + the highest overlap with other reviewers, and marked 48 of 53 findings "Major", which carries no + information. cubic had the highest share acted on and a severity mix that discriminates. +- **Gemini's findings overlap least with the others'.** It is also the only installed reviewer whose + model family is neither the primary author's nor shared with another reviewer. It raised no thread + on a workflow file in the sample, which matches its vendor's statement that it does not review + them, in an org that is mostly workflows. +- **The most structured Lens is the least acted on.** Eight code-scanning alert threads, none fixed, + three declined, four unanswered. The sample is small. It suggests the author agent treats alert + threads differently from bot review threads, which is worth a look whatever the council. +- **Two reviewers have not been reviewing.** Copilot answered 2 of 174 pull requests. Qodo's 143 + "reviews" are trial-ended notices. +- **Refusals are a design input.** Codex refused more often than it reviewed. CodeRabbit's 155 + refusals match its documented open-source limit of 1 to 10 reviews an hour per repo, against a + fleet that pushes far more often than that. + +### 16.4 SonarQube and Gitar + +**Gitar is now a Sonar product.** Sonar's product page presents it as an agent that reviews pull +requests, root-causes CI failures and commits fixes, and says it "works alongside SonarQube" with +deeper integration to come. Secondary coverage dates the acquisition to May 2026. + +| | SonarQube Cloud | Gitar | +|---|---|---| +| Kind | Deterministic rules and taint analysis over the whole project, judged on new code | An LLM reviewer plus a CI-failure agent that pushes fix commits | +| What it sees that a diff reader does not | Duplication, complexity, coverage on new lines, rules for GitHub Actions, Terraform, secrets and shell | CI logs, and whether CI goes green after its own fix | +| Structured surface | Check, annotations, issue keys through the Web API and MCP, code-scanning alerts | Review state (requests changes at a configurable threshold, approves when clean) and inline threads. No check run, SARIF or GitHub findings API is documented | +| Model | None for analysis | Not named; its bring-your-own-model page lists Anthropic models | +| Cost for public repos | Free. An open-source plan adds unlimited pull-request analysis and custom quality gates | Paid plans on the product page; its docs describe a free open-source plan granted per repo on application | +| Fit with ADR-0013 | A Lens with a full contract | **It is a writer.** Pushing commits to an agent's branch conflicts with one author per change, dismisses approvals under the ruleset, and needs write access to workflows | + +Recommendations: + +- **SonarQube Cloud: keep, and read it through its structured surfaces.** Consider the open-source + plan for custom quality gates. Its AI Code Assurance feature is a label and a stricter gate, and + its AI CodeFix repairs findings it already made; neither adds a Lens. +- **Gitar: not a council seat now.** Its review half is one more LLM on the diff with a thread-only + contract. Its CI-failure half overlaps the fleet's own failure analyst and fix-CI intent and makes + it a second author. Try it only if the in-house CI-failure path is measured as weak, with + automatic fixes off. Revisit when Sonar ships the integration it has announced, because a Lens + that combines Sonar's findings with CI outcomes through one contract would be a different thing. + +### 16.5 The wider candidate pool + +About forty candidates were examined. Three results organise the rest: + +1. **Every free, pinnable, deterministic tool publishes through SARIF or annotations.** Uploaded to + GitHub code scanning, each finding gets a stable alert identity, a dismissal state and a tool + name, free on public repos. This is the one surface available today that satisfies decision 12 + for many Lenses at once. +2. **The Lenses that differ in kind from an LLM reading a diff are deterministic analysis and + execution.** A second model brand is the weakest form of independence (§15.2). +3. **Published accuracy evidence cannot rank the LLM reviewers.** The benchmarks found are run by + vendors in the comparison, cover application code in JavaScript, TypeScript, Python or Java, and + disagree with each other. Nothing measures any reviewer on shell, YAML, Terraform or Actions, + which is most of what this org writes. + +| Candidate | What it sees | Surface | Free for public repos | Already here | +|---|---|---|---|---| +| actionlint, shellcheck, ruff | Workflow expressions, shell semantics, Python errors | Annotations; SARIF with a converter | Yes | actionlint and shellcheck run in CI and report by job result | +| zizmor | GitHub Actions security: injection, permissions, unpinned actions | SARIF, annotations | Yes | No | +| CodeQL | Dataflow security queries | Code-scanning alerts | Yes | Yes | +| Semgrep Community Edition | **The org's own rules**: each incident becomes a permanent check | SARIF | Yes | No | +| dependency-review-action | The dependency diff against advisories and licences | Job result and JSON | Yes | A dependency audit exists; not this action | +| Mutation testing on changed lines (Cosmic Ray, StrykerJS) | Whether the tests would fail if the change were wrong | Exit code and JSON | Yes | No | +| Claude Code Review (managed) | Multi-agent review with a verification step | The best-documented LLM contract: a check with annotations and a machine-readable severity line. Always neutral | No; about $15–25 a review | No | +| Greptile | Whole-repo graph; optional routing to the vendor that did not write the change | MCP with comment ids | One developer, 50 credits a month | No | +| Sourcery | An LLM on the diff | A named check that fails only on security findings | Yes, metered | No | +| Cursor Bugbot, Devin Review, Macroscope, Baz, Bito, Ellipsis | LLM reviewers | Mostly comments or a neutral check | Paid | No | +| DeepSource, Codacy, Qodana, Snyk Code | Hosted rule engines | Checks and annotations | Varies | No; same kind as SonarQube | + +No longer available as described: Korbit (acquired), Qodo Cover (unmaintained), Trunk's pull-request +checks (being retired), Amazon Q on GitHub (winding down). Google's documentation says the consumer +Gemini GitHub app was shut down in July 2026; the reviewer active here would then be the Google +Cloud version. That was not confirmed against the org's installation. + +### 16.6 A council to start measuring from + +ADR-0013 leaves composition to a follow-on record decided from each Lens's measured contribution. +This is a starting set for that measurement, chosen for independence per unit of cost and noise. + +| Seat | Lens | What it sees that the others do not | Contract surface | Status | +|---|---|---|---|---| +| 1 | Lint lane: actionlint, shellcheck, ruff | Mechanical correctness of exactly what this org ships | Code-scanning alerts, one category per tool | In CI; report as Findings instead of a job result | +| 2 | zizmor | Workflow security, in an org that is mostly workflows | Code-scanning alerts | Add | +| 3 | CodeQL, plus Semgrep CE for org rules | Dataflow, and invariants the author is not told | Code-scanning alerts | CodeQL here; add Semgrep rules as incidents occur | +| 4 | SonarQube Cloud | Whole-project gate on new code | Check, annotations, alerts, API | Here; read it by contract | +| 5 | Dependency diff | New dependencies against advisories | Job result and JSON | Partly here | +| 6 | Tests, plus mutation testing on changed lines | Whether the change's tests can fail. The nearest free thing to an acceptance signal the author cannot edit | Job result and JSON | Tests here; mutation testing to add | +| 7 | The in-house reviewer | Intent: the issue, the plan, the ADRs | The run contract | Here | +| 8 | **One or two outside LLM reviewers** | A reading by a different model, with the reviewer's own tools | Review state and threads; an API where one exists | Choose by measurement from the installed set | + +For seat 8 the installed reviewers divide like this: + +- **Keep and measure: CodeRabbit, cubic, Gemini Code Assist.** CodeRabbit is the only one with a + structured verdict, misses almost no pull requests, and runs linters in a sandbox. cubic has the + highest share acted on and the only findings API with identities and severity. Gemini overlaps + least and is the only distinct model family. Each should get a config file in the repo. +- **Keep only if its quota is funded: Codex.** A distinct model family and a good share acted on, + but it refuses more often than it reviews. +- **Remove from the required set: CodeAnt, Graphite, Copilot, Qodo.** CodeAnt adds the most volume + and the least that is unique. Graphite reviewed 22 times in a week, publishes no contract and is + not free for an org. Copilot and Qodo are not reviewing. + +That takes the outside LLM reviewers from nine to three or four, moves six deterministic and +execution Lenses onto one contract, and leaves the choice among the three to four weeks of the +Ledger's per-Lens measure. The research predicts the end state is one, at most two. + +**What the maintainer decided, 2026-10-10.** The recommendation above ranked by measured value. The +maintainer's rule is different: compatibility with the structured contract decides who stays. + +- **Off the required set now: Copilot and Qodo.** Neither is reviewing and neither shows a structured surface here. + Copilot may be re-enabled; it would then enter as a review-thread Lens under the same measure. +- **Installed, unfunded, not required: Codex.** Its refusals gate nothing and nothing waits for it. +- **Kept as review-thread Lenses: CodeAnt and Graphite.** Both open review threads, which decision + 12 names as a structured surface, so the rule does not remove them. Their threads are read by + state, open or resolved, and their summary comments gate nothing. Later the same day the + maintainer decided that only required Lenses gate, so a thread from either gates only when the + in-house reviewer adopts it. CodeAnt is free for this org. + Graphite is a paid subscription and reviewed 22 times in the measured week, which is the figure + to weigh at its renewal. +- **Kept and measured, as recommended: CodeRabbit, cubic and Gemini Code Assist.** + +That leaves six of the nine as Lenses. The per-Lens measure, not this list, decides the +required set. + +### 16.7 What is not known + +- Whether any reviewer's check conclusion can be trusted to mean "reviewed, nothing found". + Graphite's and cubic's are undocumented; CodeRabbit's is documented not to. +- Each vendor's model family where undisclosed, and so how independent CodeRabbit and CodeAnt are of + the author. +- How App-based reviewers meter or scope pull requests authored by a bot. Several budget per human + developer, which decides whether they run at all once agents have their own identity. +- Whether SonarQube Cloud's code-scanning sync will keep working here below the Enterprise plan. +- The noise of mutation testing on agent-authored changes, and any mutation tool for shell. +- Any accuracy figure for any reviewer on this org's languages. + +### 16.8 Sources + +- Org data: the weekly scorecard on `petry-projects/.github-private#1220 — 🔎 Third-Party Reviewer + Scorecard — Weekly Report` (comment of 2026-10-05); `scripts/lib/reviewer-sources.tsv`; GraphQL + reads of 2026-10-10. +- CodeRabbit: , + , + +- SonarQube Cloud: + , + , + +- Gitar: , , + +- cubic: , +- Qodo: , +- CodeAnt: +- Graphite: , +- GitHub code scanning and SARIF: + , + +- Claude Code Review: +- zizmor: ; Semgrep: ; + dependency review: + +--- + +## 17. Where a Lens runs: in the session, or on the pull request + +Some Lenses can be called from inside an agent's session before anything is pushed. Others exist +only once a pull request does. This section sorts them and asks how far checks should move left. + +### 17.1 Which Lenses can run in the session + +| Class | Lenses | In-session interface | Time | +|---|---|---|---| +| **A. Deterministic command-line tools** | actionlint, shellcheck, ruff, yamllint, markdownlint, zizmor, gitleaks, Semgrep Community Edition, the repo's own validators and tests | A local command with an exit code and JSON or SARIF output. Semgrep can report only what is new against a base commit | Seconds to a few minutes | +| **B. LLM reviewers with a local CLI** | CodeRabbit (`cr review --agent`), cubic (`cubic review --json`, `--base`), Qodo, CodeAnt, Greptile; and the fleet's own second engines, Gemini and Codex, which need no vendor | A command that reviews uncommitted or branch changes and returns findings as JSON | Minutes; CodeRabbit says to allow at least 15 | +| **C. Pull-request level only** | SonarQube Cloud's pull-request analysis, CodeQL in practice, Gemini Code Assist, Copilot code review, Graphite, Gitar, the dependency-review action, mutation testing | None usable here. SonarQube has a command-line analysis for agents that needs an entitlement this org does not have, and its snippet-analysis tool is disabled on the hosted service. CodeQL and mutation testing have command lines and take too long for a session | Minutes to hours, or server-side | + +One thing stands out in class B. **A reviewer's local CLI returns a better contract than its +pull-request surface.** CodeRabbit's agent mode emits one JSON object per finding with a severity, +a file, a line range and fix instructions; cubic's emits an `issues` array and exits non-zero when +it has findings. On a pull request the same two put severity in comment prose (§16.2). The +structured data exists; it is the pull-request channel that flattens it. + +Limits on class B: cubic's CLI needs a seat on an active subscription and is "unlimited during +alpha"; CodeRabbit's is rate-limited with a local wait of up to 15 minutes, and whether its +open-source plan covers CLI reviews is not stated. cubic's own documentation calls its pull-request +review the "final pass" because it uses a different model and pipeline from the CLI. + +### 17.2 Should checks move left? + +**Yes for feedback. No for the verdict.** + +The case for moving feedback left is the cost of a round. A finding fixed inside the session costs +one more model turn. The same finding fixed after a push costs, in the hosting repo, about 56 jobs +against 20 runner slots (§14.1), the dismissal of any approval, a fresh pass by every reviewer (four +of them produced 1,219 review events in one week), and another turn of the author. Merged pull +requests now take 7 to 10 commits at the median. Each of those commits is a round that happened on +the pull request and could have happened in the session. + +The POC has half of this already. The harness runs a lint script before committing +(`scripts/dev-lead-lint.sh`) and the prompt tells the model to run it. Three things limit it: + +- It covers shellcheck and agent-profile validation only, and describes itself as a hand-kept mirror + of the checks in `lint.yml` and `ci.yml`. +- It skips a check when the tool is missing and still exits clean. +- When the harness's own run of it fails after the model has finished, the run is aborted and the + work discarded, with a comment asking for a retry (`scripts/dev-lead-fix-issue.sh:643-668`). The + detection moved left. The repair did not. + +Four cautions set the boundary. + +1. **A check the author runs is the author's evidence.** An agent can make a check pass by weakening + it: a disable comment, an edited config, an edited test (`petry-projects/.github-private#2013 — + dev-lead fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests + to match its own code`). So each Lens runs twice. In the session its output is feedback. In + `verify`, where the author controls nothing, its output is the verdict, and only that run counts + toward a Gate. Both use the same pinned tool versions and configuration, so they agree. +2. **Whatever must stay hidden stays right.** An acceptance signal the author cannot see (PRD FR-59) + loses its value the moment it is fed back. Mutation-test survivors are fine to feed back. Holdout + scenarios are not. +3. **An outside reviewer in the session is one pass, not a conversation.** The evidence in §15.2 is + that review helps and discussion does not. A single pre-push pass by an outside reviewer may + remove several pull-request rounds; no evidence found measures it. It also spends the reviewer's + quota and the session's time budget, and the author agent's runs already time out. It should be + tried on one repo against a control before it is adopted. +4. **A git hook is the wrong mechanism for an agent.** A pre-commit hook lives in the repo the author + is editing and is skipped with one flag. The model CLI's bare mode, which its vendor says will + become the default for headless runs, does not load hooks at all. For an agent the Harness should + run the checks and return the findings for a bounded repair turn inside the same Run. For a + person, and for an interactive session, the same bundle can be an ordinary pre-commit or pre-push + hook. + +### 17.3 The shape this gives + +- **One pre-flight bundle, defined once, used in three places:** inside a Run as feedback, in + `verify` as the verdict, and in CI for changes that do not come from the factory. +- **Tiered by time.** Seconds: lint, workflow and secret checks, after the model's edits. A few + minutes: the tests for what changed, before the Result. Longer: the full suite, CodeQL, SonarQube + and mutation testing, in `verify` and CI only. +- **An inner loop with a bound.** Findings from the bundle go back to the author for at most a + declared number of repair turns. Work that still fails is kept, not discarded, and the Result + says what failed. +- **Everything in class A moves left now.** That is seats 1, 2, 3 and 5 of the council in §16.6, plus + secret scanning and the repo's tests. +- **Class B moves left as an experiment**, one reviewer on one repo, measured by rounds to + convergence and by what the pull-request pass still finds afterwards. +- **Class C stays on the pull request**, and its findings arrive through the contract of §15.5. + +The expected effect is on the constraints of §14.1 that the control plane alone does not move much: +fewer pushes per change means fewer runner jobs, fewer reviewer passes and fewer rounds for the +maintainer to watch. It does nothing for the first ceiling unless the verdict that follows can be +trusted, which is why the verdict stays where the author cannot reach it. + +### 17.4 Sources + +- CodeRabbit CLI agent mode: +- cubic CLI review: +- SonarQube CLI and MCP tools: + , + +- Semgrep CLI reference: +- Claude Code headless and bare mode: + +--- + +## 18. How it is built: in place, or greenfield + +§6.6 recommended a self-contained tree inside this repo, with a separate repo to be decided at the +end. The adversarial pass then ranked a starved migration as the likeliest failure (§14.2). This +section weighs the two ways to build and records the choice the maintainer made on 2026-10-10. + +### 18.1 The two options + +| | In-place strangler | Greenfield repo, then switch | +|---|---|---| +| Cost of each factory change | The POC's: about 56 CI jobs a push (§14.1), nine reviewers, and an approval gate that issues for none of 52 open pull requests | One hermetic required check and a council chosen from the first day | +| Risk to the running line | Shared workflows, scripts and release tags; a factory change can break the POC | None until cutover | +| "The POC takes defect fixes only" | Unenforceable in one tree | Physical: a different repo | +| Agent identity | Retrofitted under the live ruleset, with an untested rule in the way (§14.2, item 7) | The App authors agent changes from the first commit, so the maintainer can review them as an ordinary reviewer | +| Legacy doctrine | Agents working on the factory read a 1,062-line `AGENTS.md` written for the POC | A short rule file written for the factory | +| Contact with real traffic | Early and continuous | Late, unless it is designed in | +| Parity | Replaced piece by piece against the same tests | At risk: about 70,000 lines of tests encode the POC's incident history | +| Stopping | Half a strangler is hard to unwind | Archive the repo | +| Cutover | Per intent, reversible | All at once, as stated | + +Neither is right as stated. In place, the replacement is built through the line that is failing: +holding the POC still already needs an estimated 30 to 52 fix pull requests a week against 29 to +36 delivered (§14.2). Greenfield with a single switch meets real traffic only at the end, which is +how a second system fails. + +### 18.2 Two platform facts that decide the shape + +- **A literal switch is not available.** GitHub's documentation states: "GitHub Actions does not + support redirects for actions or reusable workflows." If this repo is renamed, or its name is + reused, every consumer's reference fails until it is re-pinned, across twelve repos at once. +- **Forwarding is available.** Reusable workflows may be nested up to ten levels, and one workflow + may call up to fifty. A reusable workflow in this repo can therefore become a thin caller of one + in the new repo, and permissions can only narrow down the chain. + +### 18.3 The hybrid: built greenfield, cut over as a strangler + +1. **In this repo, now:** the configuration fixes of §8, step 0, and the GitHub App identity. Both + are worth having whatever happens next. +2. **A new repo that starts as it means to go on:** a required hermetic check, the App as the + author of agent changes, environment-scoped secrets, and the starting council of §16.6, all from + the first commit. +3. **Real traffic in the first weeks.** The Kernel runs read-only from the new repo, as a poller + over every repo. It needs read access only and changes nothing. Its disagreements with what the + POC actually did are the measure of parity, and the only one a greenfield build has. +4. **Cutover at the seam that already exists.** Consumers pin this repo's channel tags. A reusable + workflow here is replaced by a forwarder to the new repo; moving a channel tag onto that commit + switches one intent for one ring; moving it back is the rollback. No consumer file changes. The + release mechanism that §3.7 lists as solid is the cutover mechanism. +5. **Last, the re-pin.** Once every intent is served by the factory at `stable`, the standards sweep + points consumers at the new repo directly, and this repo keeps only what GitHub's + `.github-private` convention gives it. + +### 18.4 What this costs, and what it does not fix + +- **Three repos for a while.** The split between this repo and the standards repo, which §3.1 + criticises, gets wider before it gets narrower. +- **Parity is a real risk.** The comparison in step 3 measures decisions, not every effect, and + some of what the POC's tests know will not be carried over. +- **It makes each change cheaper, not fewer.** The size of the build is the same either way, and + nothing in the evidence prices it (PRD OQ-9). +- **The POC must still be kept alive.** Its defect inflow does not stop because the replacement is + being built somewhere else. +- **It does not raise either ceiling of §14.1.** + +### 18.5 Conditions + +- A dated limit, a defect policy with a weekly cap for the POC, and a kill criterion, written + before the new repo is created. If the criterion is met, the configuration fixes and the identity + change stay and the new repo is archived. **Set by the maintainer on 2026-10-10**, and not as + recommended here. The limit is four weeks of side-by-side running for each intent family, counted + from the day the factory's comparable intent is available. POC defects are fixed as needed, with + no weekly cap. The kill criterion is an intent family that has not reached parity when its four + weeks end. Later the same day the maintainer replaced the parity rule, first with an outcome-based + rule and then with none: stopping is the maintainer's decision. The review intent on the + factory's own repo became the first component; ADR-0013, + decision 10, and `software-factory-design.md`, §9, are current. Nothing bounds the build before + the factory first serves a repo, which is where §14.2 expects the migration to starve. +- No intent is cut over until its disagreement list is within a bound declared beforehand and an + eval set exists for it. **Replaced on 2026-10-10:** the comparison is a measure and not a gate. + An eval set is still required. +- This repo is not renamed while anything references it. + +ADR-0013 carries these as decisions 1 and 10. The new repo is `petry-projects/software-factory`. + +### 18.6 Sources + +- GitHub, "Reusing workflow configurations": + diff --git a/docs/architecture/software-factory-architecture.md b/docs/architecture/software-factory-architecture.md new file mode 100644 index 00000000..bf008c0a --- /dev/null +++ b/docs/architecture/software-factory-architecture.md @@ -0,0 +1,135 @@ +--- +stepsCompleted: [1, 2] +inputDocuments: + - docs/prd/agentic-software-factory.md + - docs/architecture/adr/0013-factory-mvp-reference-architecture.md + - docs/architecture/factory-mvp-architecture-review.md + - docs/research/market-agentic-software-factories-research-2026-10-10.md + - "petry-projects/.github-private#2226 (discussion), analyst comment of 2026-10-10" +companionDocuments: + - docs/architecture/software-factory-design.md +workflowType: 'architecture' +project_name: 'software-factory' +user_name: 'Don' +date: '2026-10-10' +--- + +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# Architecture Decision Document + +_This document builds collaboratively through step-by-step discovery. Sections are appended as we work through each architectural decision together._ + +## Project Context Analysis + +### Requirements Overview + +**Functional Requirements:** +73 requirements in 12 groups (PRD §6). Architecturally they reduce to six capabilities: + +- A declared Lifecycle with one State per Work item, visible on the item (FR-1 to FR-7). +- One run contract in which a Role requests Effects and a separate job applies them (FR-8 to FR-14). +- One Admission point that decides from GitHub's own state, so a lost event delays work and never + deletes it (FR-15 to FR-21, FR-35 to FR-39). +- Human control that binds every Role the same way, with agents on their own identity + (FR-22 to FR-28, FR-60 to FR-63). +- A merge decision bound to content and decided from structured Findings reported by a Council + of Lenses (FR-29 to FR-34, FR-64 to FR-71). +- One Release on one gated path, with a Ledger behind every number (FR-40 to FR-59, FR-72, FR-73). + +ADR-0013's twelve decisions and the Maintainer's decisions of 2026-10-10 are fixed inputs to this +design. ADR-0000 to ADR-0012 describe the POC and are not treated as constraints. Four mechanisms +ADR-0013 takes from them are restated as this design's own rules: the thin caller, channel tags, +the optional Consumer file, and the per-item concurrency key (design document, §2.4). + +**Non-Functional Requirements:** +The ones that shape structure: + +- NFR-12, NFR-13: a Kernel free of I/O, with a size budget, running on macOS and Linux. +- NFR-14 and review §14.1: about 3 GraphQL points per evaluation, to stay under 60% of one App + installation's 5,000 requests an hour. +- NFR-1 and C-9: at most 10 Runs per merged pull request; noticing work takes at most 10% of + job-minutes from a pool of 20 runner slots shared with all CI. +- NFR-5: no Work item without a next evaluation for more than 24 hours. +- NFR-8: merge to stable within 48 hours at p90 with batched, cached evals; rollback in 15 minutes. +- NFR-4: every merge carries a standing independent approval or an Override record. + +**Scale & Complexity:** + +- Primary domain: a control plane and automation harness on GitHub Actions. No interface of its own + to operate: text on the Work item, a label vocabulary, and a read-only dashboard. +- Complexity level: high. Low data volume and no regulatory load, but very high integration + complexity, untrusted public input, self-hosting, and three scarce shared resources. +- Scale: 12 public repos, one Maintainer, about 30 to 36 merges a week in the busiest repo. +- Estimated architectural components: 15. Kernel, facts port, records port, view port, effects + port, engine port, identity port, Harness, pre-flight bundle, Controller, the brakes outside the + Kernel, release tooling, the workflow set, the simulator, and the control room. A comparator + against the POC is a later, non-gating component. + +### Technical Constraints & Dependencies + +- GitHub Free plan, every repo public, no always-on infrastructure (C-2, C-4). +- GitHub Actions starts work from events and schedules, and loses both (C-5). +- Scheduled and dispatched workflows always run the default branch, and a `uses:` reference + cannot be computed. Hence one frozen entry shim. +- Reusable-workflow references are not redirected. Cutover is pause, then repoint. +- A code owner must be a user, so the approving reviewer stays a machine user with a long-lived + token (C-6). +- LLM capacity is a subscription window of the Factory's own, with no supported usage signal + (C-3, C-11). + The credential is an environment secret in the Factory repo. The Harness calls the vendor's CLI. +- Python 3, standard library only, in the Kernel. +- Built in `petry-projects/software-factory`, which reads nothing from `.github-private` at run + time. + +Decided on 2026-10-10 and carried into the design: + +- A Finding is decided by the Role that owns its domain, under an authority table held as data. + Domain Roles start in shadow and are promoted on evidence. Only required Lenses gate. +- One GitHub App per Role, sharing the name prefix `combsmith-sf-`. +- The review Intent is built first and serves the Factory's own repo. The Maintainer approves + until the first Release on `next` serves it. +- The comparison with the POC is a measure and not a gate. Stopping is the Maintainer's decision, + if the Factory is not adopted and used. +- Each Lens declares the paths it watches. A Consumer repo's setup and test commands are part of + its onboarding entry. +- The design is modelled before it is built: schemas with golden examples, the Lifecycle as data, + a list of invariants, and a simulator that injects faults (design document, §3 to §5). + +- State is derived from GitHub at every evaluation and never stored. The record of a Run is its + Actions run record and its artifact. History is 90 days (design document, §3.9, §3.10, §7). +- Work is noticed by a poller. An event may only ask for a pass, and an optional external timer + may start one. +- Decisions read GitHub only. The control room is written and never read back. +- Every `run` job reads through one shared reader App. A Role gets its own App when it first + writes. +- `petry-projects/software-factory-control-room` is a public, rebuildable view with a dashboard, + shaped by the situation-room analysis in `petry-projects/.github-private#2242 — Situation-room + view for the agent fleet: build it from native Claude Code parts, not a command-center product` + (design document, §8). + +### Cross-Cutting Concerns Identified + +- **Credential boundaries.** No job holds both an LLM credential and a write credential; only + `verify` runs Work-item code, with no secret and no cache. +- **Schema versioning.** Every message is versioned, and a reader rejects a version it does not + know. +- **One definition of State.** The Kernel computes it, and the status on the item and the row in + the control room are the same answer. +- **One facts snapshot, three uses.** The Kernel's input is also the replay fixture and the + contract-test fixture. +- **Fail to a Hold.** An unreadable fact, a missing capacity reading or a failed validation never + yields an action. +- **Idempotent Effects.** GitHub writes carry no idempotency key, so every Effect type declares a + recovery key that is not text in a body. +- **Finding authority.** Each category of Finding has exactly one owning Role. No Role decides on + its own work. A security Finding is never declined without an Override. +- **Typing third-party prose.** Thread reviewers carry category and severity as text. They are + typed only when the in-house reviewer adopts a Finding and raises its own. +- **Brakes outside the Kernel.** The pause variable, the Run ceiling, the actor-based loop count + and, on a metered credential, a monthly cap read from the provider live where a Kernel defect cannot reach them. +- **Bootstrap.** Until a Release on `next` serves the Factory's repo, the Maintainer's attention is the line's + capacity. The first slice is cut to one event, the Factory approving one of its own pull + requests, and the approvals it takes are counted. +- **Trust classes for text.** Text from the Maintainer, a Role, an Advisory reviewer or an Outside + contributor reaches a Task as labelled data. diff --git a/docs/architecture/software-factory-design.md b/docs/architecture/software-factory-design.md new file mode 100644 index 00000000..bcadf1b7 --- /dev/null +++ b/docs/architecture/software-factory-design.md @@ -0,0 +1,2088 @@ +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# Software Factory: interaction and contract design + +**Status:** Draft for maintainer review +**Date:** 2026-10-10 +**Builds on:** [ADR-0013](adr/0013-factory-mvp-reference-architecture.md), the [PRD](../prd/agentic-software-factory.md) and the [architecture review](factory-mvp-architecture-review.md) + +## 0. The design on one page + +The Factory is a set of agent Roles that turn an issue into a reviewed, merged change across the +org's repos, and a small core that decides what each Role may do next. + +**It keeps no state of its own.** Every few minutes one job, the poller, reads every open work item +from GitHub, and reads the list of the Factory's own past Runs from GitHub Actions. From those two +things alone one pure function, the Kernel, works out where each item stands and what should happen +next. If a tick is lost, the next one reaches the same answer. + +**A Run is one workflow run with up to three jobs.** `run` gives a model a Task and gets back a +Result that only requests changes. `verify` runs the checks where the author controls nothing. +`apply` reads the item again, and only then makes the changes, through the Role's own GitHub App. +The model never holds a credential that can write. + +**A change converges when** every required check has reported, no required reviewer and no person +with write access has an open objection, and, once acceptance contracts are in force, every +criterion written before the work has passing evidence. Third-party reviewers inform the in-house +reviewer, and gate only through it. + +**People act in GitHub.** A label holds, releases or admits. A review blocks. A comment steers. The +Factory reaches the Maintainer by assigning the item to them. + +**The control room** is a public page built from the Runs' own records. The Factory writes it and +never reads it, so it can be wrong without anything going wrong. + +**It is built review-first,** in a repo of its own, served by its own newest Release. The first +milestone is the Factory approving one of its own pull requests. + +**Not yet designed** (§9.9): how an issue becomes a pull request; how a Release is cut; what the +Factory does as capacity runs short; how an interrupted Run resumes; the scorecard's sums. + +## 1. Purpose + +ADR-0013 says what shape the Factory has and why. This document says how its parts talk to each +other, precisely enough to be checked before any of it is built. It is written to be argued with: +every contract has an example, every behaviour has an invariant, and every invariant has a named +way to test it. + +It has five layers. Each is an artifact that a check can read. + +| Layer | Question it answers | Artifact | Checked by | +|---|---|---|---| +| 1. Structure (§2) | What are the parts and who calls whom? | Diagrams in this document | Review | +| 2. Contracts (§3) | What exactly crosses each boundary? | One schema and golden examples per message | Schema validation of every example | +| 3. Behaviour (§4) | What may happen, in what order, and what must never happen? | The Lifecycle as data, a list of invariants, a simulator | Exhaustive checks on the Lifecycle; seeded simulation with injected faults | +| 4. Team interactions (§5) | Which Role sees what, raises what and decides what? | Role manifests and an authority table | Mechanical checks on the data | +| 5. Platform facts (§6) | What does GitHub actually do? | A list of spikes with pass criteria | One trial each on a canary repo | + +Terms are those of the PRD glossary and are used exactly. + +### 1.1 Decisions this design takes as given + +Made by the Maintainer on 2026-10-10, in addition to ADR-0013's twelve decisions. + +| Decision | Effect on this design | +|---|---| +| The repo is `petry-projects/software-factory` | Named throughout | +| A Finding is decided by the Role that owns its domain | The authority table (§5.2) and the Finding lifecycle (§4.3) | +| One GitHub App per Role | Each Role's permissions are a ceiling GitHub enforces (§5.1) | +| The LLM credential is a subscription of the Factory's own, stored as an environment secret in the Factory repo | The engine port (§2.2); the capacity signal is observed, not read (§3.10) | +| The review Intent is built first and reviews everything after it | The build order (§9) | +| The comparison with the POC is a measure, not a gate | The comparator is a later, non-gating component | +| Stopping is the Maintainer's decision, on adoption and use | §9.2 | +| The Factory's Apps share the prefix `combsmith-sf-` | §5.1 | +| Every panel of the control room is public | §8.6 | +| Only required Lenses gate, and domain owners start in shadow (from the pre-mortem, §9.3) | §4.2, §5.2 | +| The Maintainer records continue or stop at two decision points | §9.2 | +| Decisions read GitHub only; the control room is written and never read back (from the simplification pass, §9.4) | §3.9 | +| Every `run` job reads through one shared reader App; a Role gets its own App when it first writes | §5.1 | +| A hosted relay turns webhooks and a timer into nudges; no Consumer repo carries a Factory file | §7.3 | +| Each reviewing Intent declares a rubric, which is a floor and grows from what it missed | §3.14 | +| The Kernel is written in Python, with the standard library only at run time | §7.5 | +| The Factory's own author Role comes before the control room and the shadow Roles | §9.1 | +| Each Lens has a declared scope of paths | Role manifests (§5.3) | +| A Consumer repo's setup and test commands are part of its onboarding contract | The Consumer entry (§3.12) | +| Design by simulation first, with a formal model held in reserve | §4.5 | +| State is derived from GitHub at every evaluation and never stored | §3.9, §7 | +| The record of a Run is its Actions run record and its artifact | §3.10 | +| Work is noticed by a poller; an event may only ask for a pass | §7.3 | +| The control room is a public, rebuildable view with a dashboard, holding 90 days | §8 | + +ADR-0000 to ADR-0012 describe the POC and are not constraints here. Four mechanisms that ADR-0013 +takes from them are restated in §2.4 as this design's own rules. + +### 1.2 How to read this document + +Some of it is rule and some of it is record. An implementer builds from the rules. + +| Rules | Record of how the rules were reached | +|---|---| +| §2.4, the four rules; §3, the contracts; §4, behaviour and invariants; §5, Roles and authority; §8.4, the feed; §9.1, the build order; §9.6, the serving channel and cutover | §7, options weighed and the build-or-adopt research; §8.3, what came from another analysis; §9.3 to §9.5 and §9.7 to §9.11, the review passes; §9.12 to §9.14, how it is built; §10, edits made to other documents | + +The architecture review in `factory-mvp-architecture-review.md` is history. It records +recommendations that were later replaced, and nothing should be built from it. + +### 1.3 Still open + +- Whether the POC's agents are capped while the Factory is built. They draw on the same model + window as the Factory. Uncapped for now, with the budget gauge watched. + +## 2. Structure + +### 2.1 Context + +```mermaid +flowchart LR + M[Maintainer] + O[Outside contributor] + subgraph GH[GitHub] + C[Consumer repos] + F[software-factory repo
the hub] + R[Actions run records
and artifacts] + V[(Control room
a rebuildable view)] + end + E[Engines
vendor model CLIs] + A[Advisory reviewers] + T[Deterministic tools
lint, tests, scanners] + + M -- labels, reviews, comments --> C + O -- issues, pull requests --> C + A -- checks, alerts, threads --> C + T -- checks, alerts --> C + C -- facts --> F + F -- Effects --> C + F -- every Run --> R + R -- folded by the poller --> V + V -- dashboard --> M + F -- Task --> E + E -- Result --> F +``` + +Everything the Factory knows about a Work item it reads from GitHub: the item itself, and the +Factory's own run records. Nothing authoritative lives anywhere else. The control room is a view of +those for people. The Factory writes it and never reads it, so it can be wrong, missing or rebuilt +without changing a decision. Everything the Factory does to a Work item is an Effect +applied by one job. + +### 2.2 Components + +```mermaid +flowchart TB + subgraph hub[software-factory] + shim[Entry shim
frozen, no logic] + guard[Brakes
outside the Kernel] + subgraph ctl[Controller] + pass[poller pass] + end + subgraph kernel[Kernel - no I/O] + life[lifecycle] + adm[admission] + conv[convergence and gate] + cid[content identity] + proj[projection and attention] + end + subgraph adapters[Adapters - all I/O] + facts[facts port] + rec[records port] + view[view port] + eff[effects port] + eng[engine port] + idp[identity port] + end + subgraph harness[Harness] + task[build Task] + pre[pre-flight bundle] + val[validate Result] + end + run[run job
LLM credential, read-only token] + verify[verify job
no secret, no cache] + apply[apply job
write credential, no model] + end + + shim --> guard --> pass + pass --> facts + pass --> rec + pass --> view + pass --> kernel + pass --> task --> run + run --> eng + run --> pre + run --> verify --> apply + apply --> val + apply --> idp + apply --> eff +``` + +| Component | Responsibility | May not | +|---|---|---| +| Entry shim | Run from the default branch and call the pinned Release | Contain a step or a `run:` block | +| Brakes | Human Pause, Run ceiling, loop count, and on a metered credential a monthly cap read from the provider | Import from `kernel/` or read the control room | +| Controller | One pass: read facts and run records; call the Kernel; start Runs; write the view | Decide anything itself, or read the view | +| Kernel | Every decision: Lifecycle, Admission, Convergence, the gate, content identity, what the item shows and how urgently it needs a person | Open a file, a socket, a process or the environment; read the clock except as an argument; take a comment body | +| Facts port | One snapshot of a Work item from GitHub | Return text to the Kernel | +| Records port | List the Factory's own run records and read their artifacts | Write anything | +| View port | Write the control room's files | Be read by anything that decides | +| Effects port | Apply one Effect and read it back | Run in a job that holds an LLM credential | +| Engine port | Run one vendor CLI on one Task and return its output | Hold a GitHub write credential | +| Identity port | Mint a token for the Role's App, narrowed to the Effects requested | Mint for a Role other than the one in the Result | +| Harness | Build the Task, run the pre-flight bundle, validate the Result | Interpret prose | + +Two ports isolate choices that may change: the engine port (the subscription CLI today, a metered +API later under FR-73) and the identity port (one App per Role, with the mint rule in one place). + +### 2.3 One Run, end to end + +```mermaid +sequenceDiagram + autonumber + participant P as poller pass + participant K as Kernel + participant G as GitHub + participant Ac as Actions + participant R as run + participant V as verify + participant A as apply + participant CR as control room + + P->>G: read facts for every open item + P->>Ac: list recent run records + P->>K: decide(facts, runs, capacity, now) + K-->>P: Decision: admit(role, intent) + P->>Ac: upload the pass record: facts read, Decisions made + P->>Ac: start a Run from that Decision, named for its item + R->>R: Engine turns, pre-flight, bounded repair + R-->>V: Result and commit bundle, as an artifact with a digest + V->>V: run the same checks where the author controls nothing + V-->>A: verdict and digests + A->>A: validate Result, re-derive digests + A->>G: re-read the item, refuse if held, paused, closed or changed + A->>G: apply Effects as the Role's App + A->>G: read each Effect back + A->>Ac: upload what was applied + P->>Ac: next pass: read new run records and artifacts + P->>CR: write the view + P->>G: update the status on the item if the decision changed +``` + +`verify` is a filter, not the verdict. It saves a push by catching failures first, but the code +under test runs inside it and could forge a pass, and the hub's environment is not the Consumer's. +The Consumer repo's own required checks, run on what was pushed, remain the verdict. How often the +two disagree is measured per Consumer, and `verify` is dropped where it is often. + +The `verify` steps are skipped for an Intent that changes no code. The Result artifact exists +before `apply` starts, so it is the record of what was intended (§3.6). + +### 2.4 Four rules restated + +These came from the POC's decision log. They are restated here so that this design does not depend +on that log surviving. + +1. **Thin caller.** A workflow that runs from a default branch has jobs that each `uses:` a pinned + reusable workflow and nothing else. Its only logic is a job-level `if:`. +2. **Immutable tag, moving channel.** A Release is an immutable tag. A channel is a moving tag that + points at one, moved through one path by one identity under a ruleset. Moving it back is the + rollback. +3. **No Factory file in a Consumer repo.** The poller reads every repo, and the relay of §7.3 hears + their events. Adopting the Factory changes nothing in the repo's workflows. +4. **A concurrency key per Work item, declared at job level.** One running and one pending entry + per item is the coalescing the design wants, not a defect to work around. + +## 3. Contracts + +### 3.1 The catalogue + +Every message carries `schema` (name and major version), an identifier, and the `release` that +produced it. +A reader rejects a major version it does not know. Messages are JSON. Files a person edits are shown +here as YAML because it reads more easily; in the repo they are JSON, since the standard library +has no YAML parser and nothing third-party runs in the Factory. + +| Contract | From | To | Carries | +|---|---|---|---| +| Facts | Facts port | Kernel | What GitHub holds about one Work item, as data | +| Decision | Kernel | Controller | Nothing, defer, hold, transition or admit, with a reason | +| Pass record | Controller | Runs, replay | The facts one pass read and the Decisions it made | +| Task | Harness | Engine | Everything one Run needs | +| Result | Engine | Harness | Status, requested Effects, Findings, Dispositions, usage | +| Effect | Result | `apply` | One requested change to the outside world | +| Finding | Any Lens | Kernel | One observation, with a non-text identity | +| Disposition | Author and owner Roles, inside a Result | `apply` | A proposal or a decision on one Finding | +| Run record | Actions, and the Run's own artifacts | Controller | That a Run happened, how it ended and what it produced | +| Control-room view | Controller | People, through dashboards | Every item's computed State, and what only the Factory knows, folded from run records | +| Role manifest | Factory developer | Kernel, Harness | What a Role sees, raises, owns and may do | +| Authority table | Maintainer | Kernel | Who decides each category of Finding | +| Consumer entry | Repo owner | Kernel, `verify` | How one Consumer repo is served | +| Acceptance contract | `qa-lead`, approved by the Maintainer | Kernel, `verify`, the author Role | What a change must show to be correct | +| Rubric | Factory developer, approved by the Maintainer | A reviewing Role, its evals | The least a Role must check, and how it grades what it finds | +| Lifecycle | Factory developer | Kernel | States, Transitions, Gates and actors | + +### 3.2 Nudge + +A nudge is a request for a pass now. It carries nothing. Every pass evaluates every open item, so +there is nothing for it to say, and "no event carries work" holds because there is no payload to +carry it in. + +### 3.3 Facts + +One snapshot, read in one query, of what GitHub holds. It is also the fixture format for replay and +for contract tests, so one schema has three uses. + +| Field | Content | +|---|---| +| `item`, `state`, `is_draft`, `author` | Identity, open/closed/merged, draft flag, authoring actor | +| `class` | `agent`, `maintainer`, `dependency` or `outside`, worked out by the port from the author and from whether the head is in another repo | +| `last_pusher` | The actor who pushed the head commit | +| `targets_protected` | Whether the base branch is a protected one | +| `mergeable` | Whether the head merges cleanly into the base | +| `diff_size` | Files and lines changed | +| `head`, `base`, `merge_base` | Commit hashes | +| `content_id` | Hash of the pull request's own diff against its merge base, computed by the port | +| `changed_paths` | Paths in that diff | +| `labels` | Names | +| `reviews` | Actor, state, commit, time | +| `threads` | Thread id, path, line, when it was opened, resolved, outdated, opening actor and whether that actor has write access, resolving actor | +| `checks` | Name, App, conclusion, completion time, annotation count | +| `alerts` | Tool, rule, alert number, state, severity, path | +| `queue` | Merge-queue membership | +| `actors_since_human` | Count of automated actions since the last human one, from GitHub's timeline | +| `read_at`, `quota_remaining` | When it was read, and what the installation has left | +| `complete` | For each list above, whether every page was read; and whether the diff behind `content_id` was whole | + +An incomplete read is an unreadable one. A list cut off at a page limit must never be taken to mean +"nothing more": the Kernel defers on any fact marked incomplete, and a truncated diff means a +review, never "unchanged". + +No field holds prose. Thread and comment bodies are read by the port into a separate `texts` set, +keyed by id and labelled with a trust class. The Harness may put them in a Task. The Kernel never +receives them. + +### 3.4 Decision + +```json +{ + "schema": "decision/1", + "id": "d_01JA7P", + "release": "factory/v0.3.1", + "item": { "repo": "petry-projects/software-factory", "type": "pull_request", "number": 42 }, + "outcome": "admit", + "reason": "blocking_findings_proposed", + "evidence": ["thread:PRRT_kw7", "run:r_01JA6Z"], + "admit": { "role": "pr-review", "intent": "adjudicate-findings" }, + "transition": { "from": "addressing", "to": "adjudicating" } +} +``` + +`outcome` is one of `nothing`, `defer`, `hold`, `transition`, `admit`. `reason` is an enumerated +code, never a sentence; the sentence a person reads is rendered from it. + +Every pass uploads a **pass record**: the facts it read and the Decisions it made. It serves three +purposes. Any decision can be replayed on a laptop from exactly what the Kernel saw. A Run is +started from a Decision in it, and from nothing else (§3.5). And the control room is built from it. + +### 3.5 Task and Result + +```json +{ + "schema": "task/1", + "run": "r_01JA7Q", + "release": "factory/v0.3.1", + "role": "pr-review", + "intent": "adjudicate-findings", + "item": { "repo": "petry-projects/software-factory", "type": "pull_request", "number": 42 }, + "content_id": "sha256:5d0c", + "findings": ["f_9c1e", "f_22ab"], + "texts": [ + { "id": "thread:PRRT_kw1", "trust": "advisory", "source": "coderabbitai", "body": "..." }, + { "id": "issue:body", "trust": "maintainer", "source": "don-petry", "body": "..." } + ], + "limits": { "turns": 20, "repair_turns": 2, "seconds": 900, "cost_ceiling_usd": 1.5 }, + "tier": "deep", + "permitted_effects": ["resolve_thread", "comment"], + "result_schema": "result/1" +} +``` + +A Run is started with one input: a pointer to the Decision that admitted it, in a pass record. The +Harness builds the Task from that Decision. A Run started any other way finds no Decision and does +nothing, so nothing that can merely start a workflow can choose what a Run does. + +A Task that contains text of class `outside` or `advisory` is given no shell and no network tool. +A review needs neither, and text a stranger wrote is exactly what an injection arrives in. + +Trust classes are `maintainer`, `role`, `advisory` and `outside`. Text of any class is data: it +cannot change the Task's permissions, limits or result schema. + +`cost_ceiling_usd` limits this one Run. The Harness enforces it, because it knows what its own Run +has spent. It is not an input to Admission. + +```json +{ + "schema": "result/1", + "run": "r_01JA7Q", + "release": "factory/v0.3.1", + "status": "completed", + "dispositions": [ + { "finding": "f_9c1e", "decision": { "result": "accepted" } }, + { "finding": "f_22ab", "decision": { "result": "rejected", "reason": "fix_incomplete" } } + ], + "findings": [], + "effects": [ { "type": "resolve_thread", "thread": "PRRT_kw1" } ], + "bundle": null, + "preflight": [], + "usage": { "engine": "claude-cli", "model": "recorded-by-harness", "input_tokens": 41200, "output_tokens": 2100, "cost_usd": 0.42, "seconds": 310, "session": "s_7f" } +} +``` + +`status` is `completed`, `incomplete`, `failed`, `refused` or `superseded`. A Result has no field for a commit +hash: code changes are local commits handed over as a bundle, and `apply` writes the hash after it +has pushed and read back. `usage` is filled by the Harness from the CLI's own output, including for +a Run that failed. + +### 3.6 Effect + +GitHub writes carry no idempotency key. Each Effect type therefore declares how `apply` recognises +that it has already happened. + +| Effect | Recovery key | Naturally idempotent | +|---|---|---| +| `push_commits` | The tree digest of the branch head, read back | Yes | +| `open_pull_request` | The head branch name | Yes | +| `submit_review` | A review by this actor on this commit | Yes | +| `resolve_thread` | The thread's resolved flag | Yes | +| `add_label`, `remove_label` | The label's presence | Yes | +| `create_check_run` | `external_id` set to the Effect's id | Yes | +| `enqueue_merge`, `dequeue_merge` | Queue membership | Yes | +| `assign`, `request_review` | The assignee or requested reviewer being present | Yes | +| `add_reaction` | The reaction by this App on that comment | Yes | +| `comment` | None | **No** | + +For `comment`, the Result artifact is the intent record: it lists the Effect before `apply` +starts. `apply` uploads the comment's node id after it posts. On recovery an intended comment with +no node id is resolved by listing that App's comments since the Run began, never by matching body +text. Prefer a check run to a comment wherever a person does not need to reply. + +Before any Effect, `apply` reads the item again. If it is held, paused, closed, or its `content_id` +is no longer the one the Run was admitted for, `apply` writes nothing and the Run ends as +`superseded`. A Run can wait minutes for a runner, and a hold label added in that time must still +bind. + +Two bindings stop `apply` being steered. An approval is submitted for exactly the commit of the +content the Decision admitted, never for a pull request by number alone. And the commit bundle is +taken by the identifier and digest that the `run` job reported, never by name, because the code +under test in `verify` can upload artifacts of its own. + +There is no Effect that starts a Run. Only Admission does that. + +### 3.7 Finding + +```json +{ + "schema": "finding/1", + "id": "f_9c1e", + "release": "factory/v0.3.1", + "source": { "lens": "pr-review", "kind": "role", "actor": "combsmith-sf-pr-review[bot]" }, + "surface": { "type": "thread", "key": "PRRT_kw7" }, + "location": { "path": "kernel/admission.py", "start_line": 88, "end_line": 94 }, + "category": "functional", + "severity": "major", + "adopted_from": { "lens": "coderabbit", "surface": { "type": "thread", "key": "PRRT_kw1" } }, + "content_id": "sha256:5d0c" +} +``` + +- `id` is derived from the source and the surface key. It is never derived from text, so an edited + summary is not a new Finding. +- `surface.type` is `thread`, `alert`, `annotation`, `check` or `role`. +- `category` is one of `functional`, `security`, `architecture`, `delivery`, `reliability`, + `requirements`, `mechanical`, `uncategorised`, or `human`. +- `severity` is `blocking`, `major`, `minor` or `info`. +- Category and severity are always fields. A deterministic tool supplies them. A Role supplies them + for its own Findings. A third-party reviewer that writes them only as prose gets them by + **adoption**: `pr-review` is given that reviewer's Finding as input and, if it agrees, raises a + Finding of its own with `adopted_from` set. No code parses the prose and no separate Role exists + to type it. +- A Finding has no stored status. Whether it is open is read from its surface: the thread's + resolved flag, the alert's state, the tool's latest report. + +### 3.8 Disposition + +```json +{ + "finding": "f_9c1e", + "content_id": "sha256:5d0c", + "proposal": { "outcome": "fixed", "by": "dev-lead" }, + "decision": { "result": "accepted", "by": "pr-review" } +} +``` + +A Disposition has two halves, made by two different Roles. The author proposes `fixed`, `declined`, +`deferred` or `not_applicable`. The owner of the Finding's category decides `accepted` or +`rejected`. + +It is not kept as a record of its own. Each half travels in the Result of the Run that made it, and +`apply` checks it when it acts: it will not resolve a `security` thread on a decision of declined +or deferred, whichever Role asks. What the gate reads afterwards is the surface itself. A thread +resolved by the owning Role's App is an accepted Disposition. A thread still open after the owner's +Run on this content is a rejected one. + +### 3.9 State: derived + +The State of a Work item is not stored. It is computed at every evaluation by one Kernel function +from two inputs, both held by GitHub: the facts and the Factory's run records. A stored copy could +only go stale, which is how the POC's markers came to outlive the objects they described. + +| Computed every time | From | +|---|---| +| The Lifecycle state and its reason | Facts | +| Whether the item is held, and by whom | A label, and who applied it | +| Whether a Run holds the item now | A run record that is in progress, or admitted and still waiting for a runner | +| How many attempts an Intent has had, and how many Runs the item has had | Run records since the last action by a person with write access, counting item faults only (§3.10) | +| Whether the Factory itself is failing | The outcome of its most recent failed Runs, across items | +| Whose turn it is on an open thread | Whether the author has completed a Run since the thread was opened, and whether the owner has ruled since that Run. Not "the last Run on this content": the author's answer is usually new content (§9.13) | +| Whether a Finding is open | Its thread, alert or tool report | +| Whether an owner accepted or rejected a proposal | The thread, and whether the owner's Run on this content has completed | +| Whether this content was already reviewed | The reviewer's completed Run on this `content_id`, and the review object | + +One example shows why attempts must be counted and not assumed. A pull request has a red test, and +nothing else has changed. Either nobody has tried to fix it, or three fix Runs each ran fifteen +minutes, timed out and pushed nothing. The item looks the same in both cases. The run records tell +them apart, and Actions writes a run record even when the job is killed. + +Some things only the Factory knows: what a Run cost, and the detail of what a model judged. They +stay in each Run's artifact and are shown in the control room. **No decision reads them.** One +consequence follows: money does not gate. The ceiling that places a Hold is a number of Runs per +Work item, and the Hold states the amount spent (PRD FR-39). + +### 3.10 The record of a Run + +A Run is one workflow run. Its record has two parts, both kept by GitHub. + +**The run record.** Written by Actions, whatever happens to the job. The run's name carries what +the hub's own head commit cannot: + +```text +sf petry-projects/markets#521 fix-review-findings dev-lead c:5d0c r:01JA7Q +``` + +That is the repo and item, the Intent, the Role, the short `content_id` and the Run id. Status, +conclusion, start and end times and the attempt number come from Actions. + +**Which run records count.** A pass accepts a run record only if it is a run of the Factory's own +workflow file, from the default branch, started by a schedule or a dispatch, in the Factory repo. +The name is a label for people and is never what identifies a Run. A pull request from a fork runs +its own workflow inside this repo and could give that run any name and any artifacts; it is a +different workflow file and a different event, so it is ignored. Fork workflows also need approval +before they run. + +**The artifacts.** `result.json` is the Result of §3.5, uploaded by `run` with its digest. The +commit bundle travels beside it. `applied.json` is uploaded by `apply` and lists each Effect with +what was read back. `outcome.json` is a few bytes written by a final step: how the Run ended and +whose fault it was. Nothing is handed from job to job through the log: the log is shared with +whatever the model prints, and Actions rewrites it when it masks secrets. + +A Run that dies before it uploads anything still has a run record, and counts as an attempt +subject to the fault classes below. Its +cost is not known exactly; its duration is the estimate. + +**Whose fault a failure was.** A failed Run is one of four kinds, and only the first is an attempt. + +| Outcome | Example | Counts as an attempt | Then | +|---|---|---|---| +| `item` | The Engine produced an invalid Result; the fix did not pass | Yes | The item's attempt ceiling applies | +| `capacity` | The model window ran out mid-Run | No | Every model Run is deferred until the window is expected to reset | +| `environment` | A token expired; an App was uninstalled; GitHub was unavailable | No | Several in a row pause Admission automatically, raise one alert, and clear when a probe Run succeeds | +| `superseded` | The item was held or changed before `apply` | No | The next pass decides afresh | + +A pass reads `outcome.json` only for Runs that did not succeed, which are few. A killed Run has no +outcome file; it is treated as `item` unless other Runs are failing as `environment` at the same +time. The `capacity` outcome is the capacity signal the PRD has lacked (OQ-3): observed, not +estimated. One cause must never become fifty Holds. + +Run records and artifacts are kept for 90 days, which is the Factory's own history (PRD FR-40). The +control room may keep what it has collected for longer (§7.2). + +### 3.11 Role manifest + +A domain Role as it starts, in shadow: + +```yaml +schema: role/1 +name: devops-lead +kind: shadow # author | reviewer | owner | shadow | advisor +shadows: [delivery] # becomes `owns` at promotion +lens: + scope: + paths: [".github/**", "**/*.tf", "**/*.bicep", "**/Dockerfile*", "infra/**", "scripts/ci/**"] + exclude: ["docs/**"] + always: false + raises: [delivery] +intents: + - name: review-change # would it have found something the reviewer missed? + tier: standard + permitted_effects: [] # a shadow Role writes nothing + budget_class: shadow + evals: evals/review-change + - name: adjudicate-findings # would it have decided as the owner did? + tier: deep + permitted_effects: [] + budget_class: shadow + evals: evals/adjudicate-findings +``` + +Promotion changes three lines in one pull request: `kind: owner`, `owns: [delivery]`, and an `app:` +entry naming the Role's own App, with `resolve_thread` and `comment` added to its permitted Effects +and its budget class set to `gate`. + +### 3.12 Authority table and Consumer entry + +```yaml +schema: authority/1 +max_rounds: 3 +shadow_sample: 0.25 # example: the share of items a shadow Role is run on +shadow_budget: 30 # example: sampled items before the Maintainer decides on that Role +categories: + functional: { owner: pr-review, shadow: qa-lead, may_accept: [fixed, declined, deferred, not_applicable] } + security: { owner: pr-review, shadow: security-lead, may_accept: [fixed, not_applicable] } + architecture: { owner: pr-review, shadow: solution-architect, may_accept: [fixed, declined, deferred, not_applicable] } + delivery: { owner: pr-review, shadow: devops-lead, may_accept: [fixed, declined, deferred, not_applicable] } + reliability: { owner: pr-review, shadow: sre-lead, may_accept: [fixed, declined, deferred, not_applicable] } + requirements: { owner: pr-review, may_accept: [fixed, declined, deferred, not_applicable] } + uncategorised: { owner: pr-review, may_accept: [fixed, declined, deferred, not_applicable] } + mechanical: { owner: none } # closes only when the tool stops reporting it + human: { owner: human } # closes only when that person resolves it +escalation: hold_for_maintainer +``` + +This is the starting table. A domain Role is promoted by moving it from `shadow` to `owner` for its +category, in one pull request (§5.2). + +```yaml +schema: consumer/1 +repo: petry-projects/markets +ring: next +owner_role: devops-lead +ci_workflows: ["CI"] +verify: + setup: "scripts/factory-setup.sh" + test: "scripts/factory-test.sh" + protected_paths: ["scripts/factory-setup.sh", "scripts/factory-test.sh", ".github/**", "CODEOWNERS"] +reviewers: [coderabbit, cubic, gemini-code-assist, sonarqube-cloud] +``` + +Adopting the Factory is more than this entry. The repo also needs the two scripts, a ruleset that +matches, the machine user as a code owner, and its Advisory reviewers configured. A **readiness +check** in each pass compares every Consumer repo with that list and publishes what is missing, so +a Repo owner is told and does not have to find out (§8.4). + +The entry lives in the Factory repo and rides the Release. The scripts it points at live in the +Consumer repo and are the Consumer's own. `apply` refuses a change to a protected path unless a +human transition is recorded, so an author cannot weaken the tests it is judged by. + +### 3.13 Acceptance contract + +```yaml +schema: acceptance/1 +item: { repo: petry-projects/markets, type: issue, number: 512 } +criteria: + - id: AC-1 + statement: A stall with no bookings is listed as available. + priority: P0 + check: { kind: test, ref: "tests/acceptance/test_availability.py::test_unbooked_stall_is_available" } + - id: AC-2 + statement: Booking one stall twice for the same day is refused. + priority: P0 + check: { kind: test, ref: "tests/acceptance/test_availability.py::test_double_booking_is_refused" } + - id: AC-3 + statement: No new runtime dependency is added. + priority: P1 + check: { kind: static, ref: dependency-diff } + - id: AC-4 + statement: Errors follow the booking module's existing style. + priority: P2 + check: { kind: role, ref: pr-review } +protected_paths: ["acceptance/512.yml", "tests/acceptance/test_availability.py"] +``` + +Every criterion names how it will be checked. The kinds, strongest first: + +| Kind | What it is | Strength | +|---|---|---| +| `test` | An executable check that exists, and fails, before the work starts | The author cannot argue with it | +| `scenario` | A held-out scenario the author cannot see. It lives under a different owner, where no Factory App is installed | Cannot be written to | +| `mutation` | Whether the change's own tests can fail at all | Guards against tests that assert nothing | +| `static` | A deterministic tool's report | Exact, for what it covers | +| `role` | A named Role judging the change against the statement | Only as good as that Role | +| `human` | The Maintainer's confirmation | Costs attention | + +Two protections are in play and they are different. Against **change**, a protected path is +enough, and so would be an issue, since the implementing Role's App cannot edit issues. Against +**reading**, neither is: every repo here is public, and the reader App that each Run holds is +installed across the org. Only a `scenario` needs to be unseen, which is why held-out sets are kept +under a different owner. + +The file and the tests it names are protected paths, so `apply` refuses any change to them by the +Role that implements the work. Who approved the contract is not stored: it is the +Maintainer's approving review at the commit that added the file. + +### 3.14 Rubric + +A Role that reviews declares how it judges. Without that, a Finding's severity is the model's mood, +and a Result that reports nothing cannot be told from one that did not look. + +```yaml +schema: rubric/1 +role: pr-review +intent: review-change +version: 3 +criteria: + - id: PR-1 + statement: The change does what its issue asks, and nothing the issue does not ask. + category: requirements + evidence: The acceptance contract, or the issue where there is none. + severity_if_violated: major + - id: PR-2 + statement: No code path writes outside what the Role's permitted Effects allow. + category: security + evidence: The diff. + severity_if_violated: blocking + - id: PR-3 + statement: New behaviour has a test that would fail without it. + category: functional + evidence: The diff and the test results. + severity_if_violated: major + applies_when: { paths: ["**/*.py"] } +``` + +Three things bind to it. + +- **A Finding cites its criterion.** `criterion: PR-3` is a field of the Finding of §3.7, and the + severity comes from the rubric. +- **A Result reports coverage.** For every criterion that applies: `met`, a Finding, or + `not_applicable` with a reason. A reviewing Result that omits one fails validation. +- **The Role's eval set is scored against the same rubric,** so what is measured is what runs. + +**A rubric is a floor, not a ceiling.** A Role may raise a Finding that cites no criterion. It is +marked off-rubric and counted. An off-rubric Finding that is acted on, or an escaped defect that no +criterion covered, is the evidence for adding a criterion, and that is how a rubric grows. Rubrics +are among the paths only the Maintainer approves (§5.1), because removing a criterion loosens a +gate. + +`pr-review` has a rubric from the first slice. Each domain Role gets one before it leaves shadow. + +## 4. Behaviour + +### 4.1 The pull-request Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> draft + [*] --> verifying + draft --> verifying: marked ready (human) + verifying --> addressing: a blocking Finding is open + verifying --> converged: every required Lens reported, none blocking + addressing --> adjudicating: author proposed a Disposition for each + addressing --> verifying: new content pushed + adjudicating --> addressing: owner rejected a proposal + adjudicating --> verifying: all accepted, content changed + adjudicating --> converged: all accepted, content unchanged + converged --> queued: approval issued, enqueued + converged --> verifying: new content or new Finding + queued --> merged + queued --> verifying: removed from the queue + merged --> [*] + closed --> [*] + verifying --> held + addressing --> held + adjudicating --> held + converged --> held + held --> verifying: Hold released +``` + +| State | Meaning | Next actor | Stale after | +|---|---|---|---| +| `draft` | Not ready; the Factory watches and does nothing | Human | 24 h | +| `verifying` | Waiting for required Lenses to report on the current content | Lenses, with deadlines | 30 min | +| `addressing` | Blocking Findings are open | Author Role | 30 min | +| `adjudicating` | Proposals await the owning Roles | Owner Roles | 30 min | +| `converged` | Convergence holds for this content; approval may issue | Reviewer | 15 min | +| `queued` | In the merge queue | GitHub | 15 min | +| `held` | A Hold is in place, human or automatic | As the Hold states | 24 h, or the Hold's own time | +| `merged`, `closed` | Terminal | None | None | + +Any state may move to `closed` when a person closes the pull request, and to `held` on a Hold; +those arrows are not all drawn. A +Pause is not a state: it stops Admission for a scope and leaves each item where it is. + +Human transitions are declared like any other: marking ready, converting to draft, adding or +removing a hold label, requesting changes, resolving a thread, approving, merging by Override. + +The issue Lifecycle is sketched only, because the review Intent is built first: `new`, `triaged`, +`ready`, `in_progress`, `in_review` (the pull request's Lifecycle governs), `done`, `declined`, +`held`. `done` requires a merged change with a non-empty diff that closes the issue. + +### 4.2 Convergence and the gate + +Convergence is one Kernel function of facts and run records: + +1. every Lens **required** for this class of item, this stage and these changed paths has reported + on the current `content_id`; +2. no Finding of severity `blocking` or `major` raised by a required Lens is open or proposed; +3. no `human` Finding is open. A `human` Finding is a review thread, or a review that requests + changes, from a person with write access; +4. where the item has an acceptance contract, every criterion has passing evidence on this + content (§4.8). + +A *blocking Finding*, here and in the diagram above, is one that gates: severity `blocking` or +`major`, from a required Lens or a human. + +Which Lenses are required is data. The starting set is the deterministic checks, `pr-review` and +humans. A Lens joins it only with a measured contribution (ADR-0013, decision 12). + +A Finding from a Lens that is not required is shown and measured. It is given to `pr-review` as +input. If `pr-review` adopts it, it raises a Finding of its own (§3.7), and that one gates. An Advisory reviewer that posts twenty threads therefore costs the reviewer +some reading and cannot hold a pull request by volume, which is how the POC's comment gate failed. + +`pr-review` resolves every third-party thread it does not adopt. The ruleset's requirement that +threads be resolved stays in force, so the platform keeps backing the Factory, and each such thread +gets one visible outcome. "Resolved by `pr-review`, not adopted" is then a fact on GitHub, from +which each reviewer's contribution can be counted. + +This makes `pr-review` the one funnel for every outside view, and resolving a thread is quieter +than adopting it. So the other direction is counted too: for each reviewer, and for `pr-review` +itself, how many Findings were not adopted and were later proved right by a revert or a fix in the +same place. Shadow Roles are given a sample of the unadopted Findings. + +Two rules stop this feeding back into the reviewers themselves. No Factory Finding mentions a +reviewer by name, so none is drawn into a reply. And any feature by which a reviewer approves once +its threads are resolved is switched off, in that reviewer's configuration file in the repo; +otherwise the Factory resolving a thread would hand the reviewer an approval to give. + +Model Lenses are admitted only after the deterministic required Lenses have passed on that content. +A change that fails lint is not worth a model's time, and a branch that is pushed every few minutes +settles before a review starts. + +The approval is issued only in `converged`, by the reviewer identity, for that `content_id`. If a +push changes the head and not the `content_id`, the approval is re-issued with no Run. + +### 4.3 The life of a Finding + +```mermaid +sequenceDiagram + autonumber + participant L as Lens outside the required set + participant K as Kernel + participant Rv as pr-review + participant Au as author Role + participant Ow as owner Role + participant G as GitHub + + L->>G: thread, alert or annotation + K->>G: read facts + K->>Rv: admit: review this content, with third-party Findings as input + Rv-->>K: its own Findings, some adopted + K->>Au: admit: address open Findings (one Run) + Au-->>K: proposals, plus a commit bundle + K->>Ow: admit: adjudicate (one Run per owner per content version) + Ow-->>K: accepted or rejected, per Finding + K->>G: the owner's App resolves each accepted thread + K->>K: Convergence? +``` + +- Lenses never address each other. Every arrow passes through the Kernel. +- At first the owner of every model-judged category is `pr-review` itself (§5.2). +- An owner decides in one batched Run per content version, so the number of Runs does not grow with + the number of threads. +- A shadow Role is given the same Findings on a sample of items. Its decisions are recorded in its + Run's artifact and compared with the owner's. They move nothing. +- A rejected proposal returns the item to `addressing`. After `max_rounds` the item is placed on + Hold with the open Findings listed. +- A `security` Finding cannot be accepted as `declined` or `deferred` by any Role. Only an Override + by the Maintainer does that. +- A `mechanical` Finding is never judged by a model. It closes when its tool stops reporting it. +- A `human` Finding closes only when that person resolves it. + +### 4.4 Invariants + +These are the properties the simulator asserts. Each is a sentence a test can falsify. + +| # | Invariant | PRD | +|---|---|---| +| I-1 | Every open item is evaluated on every pass, and within its state's limit once faults stop; a pass that has not run is itself reported | FR-3, FR-5, FR-43 | +| I-2 | At most one Run that can produce Effects is active per item | FR-16 | +| I-3 | Active Runs never exceed the org limit, and Runs admitted in a window never exceed the Run ceiling | FR-35 | +| I-4 | No Effect is applied twice | ADR-0013, decision 4 | +| I-5 | No Effect is applied from a Result that failed validation, or outside the Role's permitted Effects | FR-9, FR-10 | +| I-6 | No Run is admitted on a held item, and removing the Hold resumes work with no other event | FR-23 | +| I-7 | No Run is admitted under a human Pause, and automation never clears one | FR-24 | +| I-8 | Automated actions since the last human decision never exceed the ceiling | FR-17 | +| I-9 | An approval exists only for a `content_id` on which Convergence held | FR-31, FR-67 | +| I-10 | No Role decides a Disposition on a Finding about its own work | FR-30 | +| I-11 | Unchanged content causes no new review Run | FR-7 | +| I-12 | Every recorded Transition is declared in the Lifecycle | FR-1 | +| I-13 | An unreadable fact or a missing capacity reading never yields an admission | FR-35, C-1 | +| I-14 | Every cycle in the Lifecycle passes through a bounded counter or a human transition | FR-18, FR-57 | +| I-15 | Each Finding has exactly one owner and one Disposition per content version | FR-33 | +| I-16 | A `security` Finding is never closed as declined without an Override record | Maintainer, 2026-10-10 | +| I-17 | The status on an item changes only when the decision changes | ADR-0013, decision 2 | +| I-18 | A Run that ends with no Result counts as an attempt unless its fault is classed as capacity or environment, and its item is re-evaluated | ADR-0013, decision 4 | +| I-19 | No decision reads the control room: the same facts and run records give the same Decision whatever the view holds | §3.9 | +| I-20 | Incomplete facts never converge and never admit | §3.3 | +| I-21 | No Effect is applied to an item that is held, paused, closed or changed at the moment of applying | §3.6 | +| I-22 | A Run does only what a recorded Decision admitted | §3.5 | +| I-23 | A capacity or environment fault is never counted as an attempt, and one cause raises one alert | §3.10 | +| I-24 | An empty diff never converges | FR-4 | +| I-25 | An `outside` item gets no Run until the Maintainer has admitted it | FR-63 | +| I-26 | No approval is attempted where the approver is the author or the last pusher; the item is held instead | C-6 | +| I-27 | No model Lens runs on content that has not passed the deterministic required Lenses | §4.2 | +| I-28 | Attempts are reset only by an action on the item from a person with write access. A stranger's comment, an App's and the machine user's do not count | FR-17 | +| I-29 | Once acceptance contracts are in force, no implementing Run is admitted for an item without an approved one | FR-59 | +| I-30 | No change by an implementing Role touches an acceptance contract or a test it names | FR-59 | +| I-31 | An item with an acceptance contract never converges with a criterion lacking passing evidence on that content | §4.2 | +| I-32 | While more items need the Maintainer than the declared number, no Run is admitted except one that unblocks a merge | C-10 | +| I-33 | Every write on GitHub by a Factory identity matches an Effect that `apply` recorded; one that does not raises an alert | §4.9 | +| I-34 | A run record counts only if it is the Factory's own workflow, on the default branch, started by a schedule or a dispatch | §3.10 | +| I-35 | A Run whose Task holds `outside` or `advisory` text has no shell and no network tool | FR-12 | +| I-36 | An `outside` item's admission holds only if it was given after the author's latest push | FR-63 | +| I-37 | A reviewing Result reports on every criterion of its rubric that applies; one that does not applies no Effect | §3.14 | +| I-38 | A Run asks the Kernel again as it starts, and does nothing if it is no longer the Run that is wanted | §9.13 | + +### 4.5 The simulator + +The Kernel does no I/O and takes the clock as an argument. That makes it possible to run the real +decision logic against a fake world, quickly, thousands of times. + +- **World.** An in-memory GitHub behind the facts, records and effects ports: repos, items, content + versions, actors, reviews, threads, checks, labels, a merge queue, and run records with artifacts. It applies GitHub's own rules, + including dismissing an approval when the merge base changes. +- **Actors.** Scripted stand-ins for an author that sometimes fixes and sometimes breaks, Lenses + that raise Findings, owners that accept or reject, and a Maintainer who labels, reviews and + merges at arbitrary moments. +- **Clock and scheduler.** Time advances only when the simulator says. It decides which tick or + nudge is delivered next. +- **Faults.** A tick dropped or delayed by hours. A nudge dropped or duplicated. A job killed at + every step boundary, including between two Effects and before any artifact is uploaded. The + control room unwritable, or holding nonsense. A facts read that fails or is stale. A missing capacity reading. + A Result that fails validation. An Engine that times out. A human acting mid-Run. +- **Properties.** I-2 to I-17 are checked after every step, as are I-19 to I-38. I-1 and I-18 are checked once faults + stop and the world is left to settle. +- **Reproduction.** Each run has a seed from the standard library's generator. A failure prints the + seed and the trace, and the trace renders as a sequence diagram. +- **Determinism.** The required check runs a fixed list of seeds, so it cannot fail at random and + stall every merge. Exploratory seeds run on a schedule and update one tracking item. A seed that + finds a fault is added to the fixed list with its fix. +- **Further faults,** from the failure-mode pass: a facts read cut off at a page limit; a hold added + between Admission and `apply`; a Run started with no Decision behind it; every Run failing at + once for one outside cause. + +- **Grounding.** The fake GitHub is only as good as what it assumes. Each rule it applies cites a + response recorded from the real platform in a trial of §6. A rule with no recording is listed as + unverified, and `factory / verify` prints that list. The POC's tests passed against a stub that + accepted calls the real interface rejects; this is the guard against repeating it. + +Two things are checked exhaustively rather than sampled, because they are small and finite: the +Lifecycle graph (every state reaches a terminal one, every non-terminal state has a timer, I-14) +and the authority data (§5.4). + +The simulator samples interleavings. It does not prove their absence. A formal model in TLA+ would, +for a small model, and is held in reserve, written in Quint (§7.5), for crash recovery if the simulator finds a +fault there that it cannot pin down. + +### 4.6 Boundary cases + +A sweep on 2026-10-10 ran the Lifecycle against awkward inputs. Each case below now has a rule. + +| Case | Rule | +|---|---| +| A pull request from a fork, or from someone who is not a collaborator | Its class is `outside`. It gets no Run, because a Run holds the model credential. It is held as "needs you" until the Maintainer admits it or closes it (I-25). Its status is written for a stranger: what is happening, and that a maintainer has to admit it. Admission lapses when the author pushes again, so a harmless commit cannot be admitted and then replaced (I-36) | +| A dependency update | Its class is `dependency`. Its required Lenses are deterministic ones. The POC's automerge serves it until that Intent is cut over | +| The approver is the author or the last pusher | GitHub rejects or discards the approval. The Kernel reads `author` and `last_pusher`, attempts no approval, and holds the item with that reason (I-26). The machine user therefore never pushes (§5.1) | +| A branch pushed every few minutes | Deterministic Lenses first; a model review starts only on content they have passed (I-27) | +| A Hold released, or a pull request reopened, after attempts ran out | Attempts count since the last action by a person with write access, which grants a fresh allowance. No agent's action does, and nor does a stranger's comment: on a public repo that would let anyone keep an agent running (I-28) | +| An empty diff | Never converges (I-24). The POC merged ten | +| A diff too large to hash or to review | Each class has a size limit. Over it, one Hold, with the choices: split it, or review it by hand. No Run is spent finding out | +| A conflict with the base | `mergeable` is false. It is the author's turn; in the first slice it is the Maintainer's | +| A pull request that targets another feature branch, as stacked pull requests do | The Factory serves pull requests that target a protected branch. Others are shown as not served | +| The poller was down for hours | An item is stale only if it went unevaluated while passes were running. A missing pass is one health alert, not one alert per item | +| A third-party thread the reviewer does not adopt | The reviewer resolves it (§4.2), so the ruleset does not block the merge on it | +| A review thread from someone without write access | On a public repo anyone can open one. It is treated as a third-party reviewer's: `pr-review` adopts it or resolves it. Only a person with write access can hold a pull request with a thread | +| A plain comment from the Maintainer | It steers and gates nothing. It goes into the next Run's Task, and the Role's App reacts to it to show it was read. A plain comment has no resolved state, so it could never be closed. To block, use a review | + +Already handled with no special rule, because State is derived: a draft, a conversion back to +draft, a reopened pull request, a force-push during a Run (`apply` re-reads the item), ejection +from the merge queue, and an early approval by a person. + +### 4.7 The control vocabulary + +Every human control is one ordinary GitHub action, the same for every Role and every item. + +| Control | What the Maintainer does | Effect | +|---|---|---| +| Start work on an issue | Add the label `factory:ready` | The issue comes under Factory control | +| Hold | Add the label `factory:hold` | No Run is admitted on the item; an in-flight Run writes nothing (I-6, I-21) | +| Release | Remove the label | The next pass resumes, with a fresh allowance of attempts (I-28) | +| Hands-off | Add the label `factory:hands-off` | The Factory watches the item and never acts on it | +| Admit an outside item | Add the label `factory:admit` | The item is served like any other, until its author pushes again (I-25, I-36) | +| Steer | Write a plain comment | It reaches the next Run's Task and is acknowledged with a reaction | +| Block | Open a review thread, or request changes | A `human` Finding, which gates until that person resolves it | +| Approve | Approve the pull request | A human approval stands as an independent approval | +| Override | Merge through GitHub's own bypass | The merge is recorded as an Override and a reason is asked for on the item | +| Pause | Set the variable `FACTORY_PAUSED` for the org or a repo | No Run is admitted in that scope; only a person clears it | +| Re-evaluate | Nothing | Every pass evaluates every open item | + +A pass reads only the issues that carry `factory:ready`, so its cost follows the work in hand and +not the size of the backlog. The rest of the issue Lifecycle is not yet written. + +The Factory reaches the Maintainer the same way, through GitHub. When an item becomes "needs you", +the Controller assigns it to the Maintainer, or requests their review on a pull request, and +GitHub's own notifications carry it to email and phone. A fault of the whole Factory goes to one +tracking issue, never to one per item. The Maintainer is notified once per item per reason, and +not again while that reason stands. + +**Backpressure from the Maintainer's queue.** When more than a declared number of items need the +Maintainer, Admission admits only Runs that unblock a merge. The starting number is 10. The +Maintainer's attention is the first limit on output (PRD C-10), and a factory that keeps producing +work its one human cannot clear only lengthens the queue (I-32). + +### 4.8 Acceptance: what a change is judged against + +Conditions 1 to 3 of Convergence say that nobody required has an open objection. That is the +absence of objection. It is not evidence that the change is right. The acceptance contract of §3.13 +supplies the other half: a statement, written before the work, of what the change must show. + +How correctness is covered: + +| Question | Answered by | +|---|---| +| Was the work actually done? | Claims generated by `apply`; an empty diff never converges | +| Did it break what worked? | The repo's tests, its required checks, the merge queue | +| Is it safe and well built? | Deterministic scanners; domain Roles once promoted | +| Is the judge any good? | Eval sets, shadow comparison, escaped defects | +| **Did it do what was asked?** | **The acceptance contract** | +| **Does the new behaviour work?** | **Checks of kind `test`, `scenario` and `mutation` in that contract** | + +The flow is one pull request, with two authors in sequence on separate files: + +1. An issue states the intent. +2. `qa-lead` makes the first commit on the work's branch: the contract, and failing tests for the + criteria of kind `test`. +3. The Maintainer approves at that commit. This is what arming the work means: one approval, of a + concrete thing. GitHub dismisses the approval when the next commit arrives; the review remains + on that commit, and that is the record. +4. The implementing Role adds its commits to the same branch. `apply` refuses any change to the + contract or to a test it names. +5. `verify` and the Consumer's own checks run them. The Kernel reads one result per criterion. + +Failing tests therefore never reach the default branch, where they would break every other pull +request's checks and the merge queue. "One author per change" holds file by file: no file in the +pull request has two authors. + +A criterion that cannot be met is waived only by the Maintainer, as a recorded human transition. + +**Who drafts it.** `qa-lead` wraps the vendored BMAD test-architecture agent, whose workflows are +built for this sequence. Three are used and one is not: + +| Workflow | Used for | +|---|---| +| Test design | The priority of each criterion and the kind of check it deserves | +| ATDD | Failing acceptance tests written before the implementation | +| Trace, first phase | The map from each criterion to the tests that cover it | +| Trace, second phase | **Not used.** It has a model issue a PASS or FAIL verdict. Here the Kernel decides, from one result per criterion | + +Three things have to be built for that to work. The workflows are interactive and produce +documents for people, so the Intent needs a wrapper that runs them in one Run and returns the +contract as data. `qa-lead` becomes a Role that writes code, so it needs an eval set and, when it +first writes, its own App. And the agent starts from criteria it is given: it turns them into +checks and says when one is untestable, but what the work is for comes from the issue. + +**Independence of the drafter.** The Role that drafts the contract and the Role that implements it +are different, but models that are wrong tend to be wrong in the same way. A test can then encode +the same misreading as the code it tests. Where the engine configuration allows, the drafting +Intent is served by a different model family from the implementing one. This is the place in the +Lifecycle where a second family buys the most, and it depends on that vendor's quota. + +**How strict, and when.** The contract takes effect with the fix and implement Intents (§9.1, +slice 2). At first the rule is only that every criterion names a check and has evidence, and a +`role` check is allowed. It tightens as measurements arrive: `P0` criteria must have an executable +check, then held-out scenarios where they prove their worth. + +For the Factory's own repo the contract exists already. The invariants of §4.4 and the simulator +of §4.5 were written before the code, and they are among the paths only the Maintainer may approve +(§5.1). + +What this does not fix: criteria that are themselves wrong; the Maintainer's attention, since each +contract is one more approval; a `role` check, which is model judgement; and mutation testing, +whose noise on agent-written changes nobody has measured and for which no shell tool exists. + +### 4.9 Threats, and the rules that answer them + +A security pass on 2026-10-10 read the design as an attacker with no write access, as a defender +and as an auditor. Everything here is public: repos, run records, logs, artifacts and the control +room. That shapes every row. + +| Attack | Rule | +|---|---| +| Inject text the model reads, and have it print the model credential, encoded, into a public log. No network access is needed | Three layers, none sufficient alone. No shell or network tool for a Run that reads untrusted text (I-35). The model CLI's sandbox, if a trial shows it can hide the parent's environment and limit connections (P-26). A credential that belongs to the Factory alone, so that theft is a rotation and not the loss of the Maintainer's own account | +| From a fork, start a workflow run named like a Factory Run, with forged artifacts | Run records count by workflow, event and repo, never by name (I-34) | +| Put instructions in a pull-request title, which reaches the feed, which a board in the Maintainer's own Claude session reads | The feed carries no free text from outside (§8.4) | +| Get a harmless fork commit admitted, then push the real one | Admission lapses on the next push (I-36) | +| Steal what the relay holds | It holds a webhook secret and nothing that can act on the Factory repo. The worst a thief can do is cause passes (§7.3) | +| Forge a fault report from inside a Run, to pause the Factory | A pause needs agreeing faults from Runs on different items, and `apply` and the poller class the faults they see themselves. The worst case is delay, bounded by the probe Run | +| Replace the commit bundle from inside `verify` | The bundle is taken by identifier and digest (§3.6) | +| Craft a Result that has `apply` approve something else | An approval is for exactly the admitted commit (§3.6) | +| Put markup in a title to script the dashboard | The page renders text as text and builds links only from identifiers | +| Use an expression in a workflow to run text as a command | No workflow passes data into a `run:` block by expression; the workflow-security Lens checks it | + +**Reconciliation.** The most useful control is a detective one, and the design already records +what it needs. Each pass compares what the machine user and each App actually did on GitHub with +the Effects `apply` recorded. A write with no matching Effect means a credential is being used +outside the Factory, and raises one alert (I-33). This matters most for the machine user's token: +it is long-lived, it can approve, and it can merge. + +**Credentials.** + +| Credential | Can | Held by | Stored | Rotated | +|---|---|---|---|---| +| The Factory's model subscription | Spend the Factory's model capacity | The `run` job | Environment secret, default branch only | On any suspicion; it is the Factory's alone | +| The machine user's token | Approve and merge, across the org | `apply`, for the review Intent only | Environment secret, default branch only | On a fixed period, and on any reconciliation alert | +| Each Role's App key | Mint that Role's tokens | `apply` | Environment secret, default branch only | On a fixed period | +| The Controller's App key | Start Runs; write the control room, statuses and labels | The poller | Environment secret, default branch only | On a fixed period | +| The reader App's key | Read public repos | The `run` job | Environment secret, default branch only | On a fixed period | +| The held-out sets' credential | Read the held-out repo | The eval job only | Environment secret, default branch only | On a fixed period | +| The relay's webhook secret | Prove that a webhook came from GitHub | The relay | The relay's host | On a fixed period | +| The relay itself | Nothing over the Factory repo | The relay's host | None needed | Not applicable | + +If the subscription cannot be used, the fallback is a metered API credential under the monthly cap +of PRD FR-73, held and stored the same way. + +Only the Controller can start or delete runs. The readiness check (§3.12) asserts the settings +these rules rest on: fork workflows need approval, the default workflow token is read-only, secret +scanning with push protection is on, and environments are restricted to the default branch. It +also records what each third-party reviewer has been granted, since several can write to branches, +and reports any change. + +## 5. Team interactions + +### 5.1 Roles and identities + +Each Role acts through its own GitHub App, so its permissions are a ceiling the platform enforces. +A token is still narrowed at mint time to the Effects one Result requests. + +| Role | At the start | Owns at the start | Owns once promoted | Writes as | +|---|---|---|---|---| +| `pr-review` | Reviewer, and owner of every model-judged category | `functional`, `security`, `architecture`, `delivery`, `reliability`, `requirements`, `uncategorised` | `requirements`, `uncategorised` | Its own App: comment, open and resolve threads. Its approval is submitted by the machine user | +| `dev-lead` | Author, from the fix Intents onward | None | None | Its own App: push to work-item branches, open pull requests, comment | +| `qa-lead` | Shadow; from the fix and implement Intents, author of acceptance contracts | None | `functional` | Its own App when it first drafts a contract | +| `security-lead` | Shadow | None | `security` | Nothing until promoted; then its own App | +| `solution-architect` | Shadow | None | `architecture` | Nothing until promoted; then its own App | +| `devops-lead` | Shadow | None | `delivery`, and Consumer entries | Nothing until promoted; then its own App | +| `sre-lead` | Shadow | None | `reliability` | Nothing until promoted; then its own App | +| `business-analyst` | Advisor on issues | None | None | Its own App when it first comments | +| `scrum-master` | Advisor on planning | None | None | Its own App when it first comments | +| Controller | Not a Role | None | None | Its own App: the control room, the status on the item, labels and the queue | +| Reader | Not a Role | None | None | Nothing. Every `run` job reads through it | + +This table is generated from the manifests and the authority table, so it always shows the present +state. The fourth column is the plan. + +- **Apps are installed across the org, and tokens are minted for one repo.** A new Consumer repo + then needs no installation step, and a token cannot reach a repo its Run is not about. +- **Reading is shared; writing is not.** Every `run` job reads GitHub through one App, + `combsmith-sf-reader`, which can write nothing. A Role is given an App of its own when it first + needs to write, and `apply` mints that App's token. A Role in shadow writes nothing, so it runs + on the reader alone until it is promoted. +- The approving review stays with the existing machine user, because a code owner must be a user. + It approves and does nothing else. It never pushes and never opens a pull request, because GitHub + will not count an approval from an author or a last pusher. +- **The Maintainer is the only code owner for the Factory's own rules** in the Factory repo: + `lifecycle/`, the authority table, the rubrics, the gate function, the brakes, the invariants and + the simulator's fixed seeds, and `.github/`. GitHub then + enforces that the Factory cannot approve a change to what governs it, whichever Release is + serving. +- The Apps share the name prefix `combsmith-sf-`, so `qa-lead` acts as `combsmith-sf-qa-lead[bot]`. The frozen Consumer file drops + Factory-authored events by that prefix, so adding a Role never changes a Consumer file. +- Registering an App is a manual step. "A new Role in one pull request" (NFR-11, UJ-7) becomes one + pull request plus one registration. A new Intent or trigger for an existing Role is still one + pull request. + +### 5.2 Who decides what + +The authority table of §3.12 is the whole rule. Five points follow from it. + +- **Ownership is by category, not by path.** A security defect can sit in any file. +- **An owner judges; it does not rewrite.** There is one author per change. +- **A domain Role starts in shadow.** `pr-review` owns every model-judged category at first. On a + sample of items a domain Role both reviews the change and adjudicates the same Findings, on the + reader App. What it would have raised and decided is recorded, and binds nothing. +- **A domain Role is promoted on evidence, one Role at a time.** It needs an eval set for its + adjudicating Intent (FR-55) and a record in shadow that shows it adds something: agreement with + the owner where the owner was later proved right, and disagreement where the owner was later + proved wrong. The research is plain that a reviewer no stronger than the author makes the result + worse, so promotion is never by default. +- **Shadow has a budget.** The evidence for promotion is escaped defects, which are rare, so a Role + could sit in shadow indefinitely. After a declared number of sampled items the Maintainer decides + for that Role: promote it, keep sampling, or stop. +- **The security rule holds from the first day.** Whoever owns the category, a `security` Finding + is never accepted as declined or deferred without an Override. + +Once promoted, an owner is measured like any other Lens (FR-69): the share of its acceptances later +reverted or reopened is the number to watch. + +### 5.3 What each Role watches + +A Lens has a declared scope of paths in its manifest. The scope is an input to Admission: the +Kernel compares it with `changed_paths` and admits the Lens only where they meet. + +| Role | Scope | Why | +|---|---|---| +| `devops-lead` | Workflow files, Terraform, Bicep, Dockerfiles, CI scripts | File type is a good proxy for this domain | +| `qa-lead` | Source and test paths; not documentation | A proxy that is mostly right | +| `security-lead` | Always, on any code change | A security defect is not a file type | +| `solution-architect` | Signals, not types: a new top-level directory, a dependency manifest, a schema, a public interface, a decision record, a diff over a declared size | Architecture is cross-cutting | +| `sre-lead` | Runtime configuration, error handling, observability | Narrow by design in the MVP | +| `pr-review` | Always | It reads the change against its issue | + +Four rules keep a scope from becoming a blind spot. + +1. A scope is not a workflow `paths:` filter. That would put logic in YAML and stop a required + check from reporting. +2. A Lens whose scope does not match is **not required** for Convergence on that content. It is + never "awaited". +3. A scope narrows what a Role *raises*. It does not narrow what it *owns*: a `delivery` Finding + raised by another Lens on any path still goes to whoever owns `delivery`. +4. A scope is measured. A Finding of a Role's category raised by another Lens on a path outside + that Role's scope is counted as a scope miss. + +### 5.4 Mechanical checks on the team + +Run by `factory / verify` over the manifests and the authority table. + +- Every category has exactly one owner. +- No Role is both the author of a change and the decider of a Disposition on it. +- No Role both proposes and decides the same Disposition. +- Every permitted Effect of every Intent is within the declared ceiling of that Role's App. +- No manifest names another Role's output as a trigger. +- The graph "a Run of X can lead the Kernel to admit a Run of Y" has no cycle that does not pass + through `max_rounds` or a human transition. +- Every Intent in budget class `gate` has an eval set. +- A Role named `owner` of a category has a promotion record. A `shadow` Role's decision is never + read by the gate. +- Every Consumer entry names an owner Role, a setup command, a test command and protected paths. + +The interaction matrix in §5.1 is generated from this data in the Factory repo. A stale copy fails +the check. + +## 6. Platform facts to settle by trial + +None of these can be modelled. Each is one trial on a canary repo. Each leaves the real response +it observed as a recorded fixture, which is what the simulator's fake GitHub is grounded in (§4.5). + +| # | Question | Pass | +|---|---|---| +| P-1 | Does a pull request authored by an App need an extra approval under the live ruleset's rule for unattributed changes? | One approval suffices | +| P-2 | Can an App enable auto-merge and enter the merge queue? | Yes | +| P-3 | Is an environment secret restricted to the default branch unreadable from a workflow on a work-item branch? | Unreadable | +| P-4 | Can an installation token be minted with a subset of the App's permissions and repos? | Yes | +| P-5 | Can one App resolve a review thread opened by another App or by a third-party reviewer? | Yes, with pull-request write | +| P-6 | Does a job-level `if:` on the actor's prefix drop events authored by the Factory's Apps? | Dropped, with no step run | +| P-7 | Does the machine user's approval satisfy code-owner review and the last-push rule when an App pushed last? | Yes | +| P-8 | What does one pass over every open item in twelve repos cost in GraphQL points, and how often do scheduled ticks arrive in the new repo? | Within the budget of §7.3 | +| P-9 | Can a pass start the next one with the default workflow token? | Yes | +| P-10 | Does a nudge count against the content-creation limit, and can its credential be dispatch-only? | Does not count; can | +| P-11 | Can the Controller's App write the control-room repo, and publish its page, without starting a workflow in the hub? | Yes | +| P-12 | Does a check run's `external_id` round-trip, so it can serve as a recovery key? | Yes | +| P-13 | Does the vendor CLI return validated structured output and usage in headless mode on the subscription credential, at a pinned version? | Yes | +| P-14 | Can a job with no permissions and no cache action write the default-branch cache scope? | Cannot | +| P-15 | Do the retained Advisory reviewers review a pull request authored by an App? | Each one does, or is not counted as a Lens on such pull requests | +| P-16 | How long are run records, logs and artifacts kept in a public repo? | 90 days | +| P-17 | Is a run's name set when the run is created, and visible for a run whose job never started? Can recent runs of one workflow be listed and matched by name within the request budget? | Yes to both | +| P-18 | Does a final step still run, and upload partial usage, when a job times out or is cancelled? | Yes, or duration is the only estimate | +| P-19 | Can a static page in a separate public repo be published from files the Controller writes? | Yes | +| P-20 | Does an event that anyone may cause, such as starring a repo, still start a workflow, and how often may it? | Yes, often enough for a tick every minute or two | +| P-26 | Can the model CLI's sandbox stop a tool reading the parent process's environment, and limit where it connects? | Yes to both, or the first and third layers of §4.9 carry the weight | +| P-27 | Does a fork's pull-request workflow run wait for approval, and can it never be a schedule or dispatch run? | Yes to both | +| P-31 | Does the Controller App's webhook reach a hosted function for every repo it is installed on, and can that function cause a nudge with no credential over the Factory? | Yes to both | +| P-28 | Can the actor who last pushed a pull request's head be read as a fact? | Yes, or I-26 has to be met another way | +| P-29 | Is the hash of a pull request's own diff stable when only the base moves, and how does a truncated diff present? | Stable; truncation is detectable | +| P-30 | Which identity can cut a Release and move a channel tag under a tag ruleset, from a workflow in the Factory repo? | One identity, through one path | +| P-21 | How often may the control room's page be rebuilt, and can a page read the feed files directly from a browser? | The page reads the feed directly and rebuilds only when its own code changes | +| P-23 | How many threads may one App resolve in an hour before GitHub's secondary limits apply? | Enough for about 22 threads a pull request at the Factory's merge rate | +| P-24 | With the Maintainer as the only code owner of a path, does the machine user's approval fail to satisfy code-owner review there? | It fails to satisfy it | +| P-25 | Does resolving a reviewer's thread cause that reviewer to re-post it, reply, or approve? | No, with the configuration of §4.2 | +| P-22 | How soon after it is started does a Run appear in the list of run records? | Soon enough that a pass never admits the same Run twice; the item's lane and I-21 cover the gap | + +### 6.1 The order to run them in + +An assumption audit on 2026-10-10 ranked each trial by what breaks if its answer is no. + +| Batch | Trials | If the answer is no | +|---|---|---| +| **1. Before any code** | P-13 (the model CLI returns structured output on a subscription); P-1, P-7, P-24 (the approval mechanics); P-17, P-22 (run records can be identified); P-3 (secrets are isolated by environment); P-28, P-29 (last pusher and content identity are readable); P-8 (a pass fits its budget) | The design does not work as written. Each of these is a foundation: no Run, no approval, no derived State, no credential boundary, or no poller | +| **2. Before the first approval** | P-9, P-20 (chaining and the outside tick); P-5, P-23, P-25 (resolving others' threads); P-2 (an App in the merge queue); P-30 (cutting a Release); P-27, P-14, P-4 (forks, caches, narrowed tokens); P-26 (the sandbox) | One mechanism is replaced and the design stands. If threads cannot be resolved by an App, required thread resolution is dropped in repos the Factory serves | +| **3. Before the control room** | P-11, P-19, P-21 (publishing the page); P-16, P-18 (retention and final steps) | The page is published another way, or history is shorter | +| **4. As they come up** | P-6, P-10, P-12, P-15 | A detail changes | + +Two assumptions are not trials. Whether a subscription used only by automation is within the +vendor's terms is a question for the Maintainer and the vendor. And the numbers this document +gives as starting figures (40 approvals, 10 items in the Maintainer's queue, a shadow budget of 30, +a sample of a quarter, three rounds, the staleness limits) are placeholders to be calibrated from +the first weeks of data. + +## 7. State, records and noticing work + +ADR-0013 first adopted a git-backed store and a wake per event, both provisionally. The Maintainer +replaced both on 2026-10-10. This section records what was decided, why, and what was set aside. + +### 7.1 Why there is no store + +The first design kept a State record per Work item and an append-only Ledger in a git store that +only the Controller wrote. The question that undid it was simple: why can State not be determined +from the issue and the pull request? + +Mostly it can. The Lifecycle state, the Hold, the thread and alert Findings and the reviews are all +on GitHub, and a stored copy of any of them is a copy that can go stale. What cannot be read from +the item is a small residue (§3.9): that a Run was attempted, what it cost, and what a model +judged. GitHub already keeps the first of these, because every Run is a workflow run, and Actions +records a run even when its job is killed. The other two are in the Run's own artifact. + +So nothing needs a second authoritative store. That also answers the adversarial review, which +faulted the first design for keeping control state that could not be rebuilt, where every +comparable system rebuilds its state from the platform. + +### 7.2 Where each thing lives + +| Need | Where it lives | Written by | +|---|---|---| +| Everything about the Work item | The issue and the pull request | People, Lenses and `apply` | +| That a Run happened, how it ended, how long it took | The Actions run record, named for its item | Actions | +| What a Run produced: the Result, its cost, what a model judged | An artifact with a digest | `run` and `apply` | +| The handoff from `run` to `verify` to `apply` | The same artifact, verified by digest | `run` | +| A fast index of the last two, and every item's computed State | The control room (§8) | The Controller | + +GitHub is the source of truth for all of it. The control room is a materialized view: it holds +nothing GitHub does not for the 90 days GitHub keeps run records, and for that span it can be +rebuilt. It may keep what it has collected for as long as it likes. Older history then exists only +there, and is used for reporting and never for a decision. + +What this costs: + +- **A killed Run's cost is unknown.** The vendor CLI reports usage when an invocation ends. + Duration is the estimate, and a final step uploads whatever partial usage exists. +- **History is 90 days.** The Maintainer accepted this in place of the 180 the PRD first asked for. + Aggregates last longer: the control room's own git history keeps every scorecard it has held. +- **Deleted run records undercount attempts.** A Run that wrote nothing leaves no other trace. + Deleting them needs write access to the Factory repo, and the count of automated actions on the + item catches every case where a Run did write. This is accepted and not designed for. +- **Money does not gate.** No decision reads the control room, and cost is known only there. The + ceiling that places a Hold is a number of Runs per Work item; the Hold states the amount spent. + A Run still stops itself at its own cost limit. On a metered credential a monthly cap is a brake + read from the provider's own usage figure, which is exact and outside the Factory (FR-73). +- **An issue with no pull request has no Run on it yet,** so there is nothing to record until the + first Run, which is then a run record like any other. + +### 7.3 Noticing work + +The Controller has to evaluate an item when something about it changes and when time passes. +GitHub Actions offers an event, which is lost if its run is evicted, and a schedule, which arrives +late or not at all. + +| # | Noticing must | From | +|---|---|---| +| W-1 | Be correct whichever single event or tick is lost | FR-5, FR-19 | +| W-2 | Reach a decision within 15 minutes at the 95th percentile after a change that produces no event | ADR-0013, decision 11 | +| W-3 | Use at most 10% of the org's job-minutes, from a pool of 20 slots shared with all CI | PRD §7, C-9 | +| W-4 | Stay under 3,000 GraphQL points an hour at peak | ADR-0013, decision 11, NFR-14 | +| W-5 | Keep one writer per item and coalesce requests that arrive during a Run | FR-16 | +| W-6 | Put no file and no credential in a Consumer repo | FR-51 | +| W-7 | Be unable to feed itself without bound | FR-18 | +| W-8 | Notice that CI has finished | Review §14.2 | +| W-9 | Need no always-on service | C-4 | + +| Option | How it works | Meets | Fails or strains | +|---|---|---|---| +| 1. A wake per event | A file in each Consumer forwards every subscribed event to the hub, which starts one Controller run for that item | W-2 at its best, W-5, W-9 | W-3: about three jobs per event, and at least 12,500 events a week from the busiest repo. W-1 still needs a sweep. W-7: the Factory's own writes are events. W-8: a `workflow_run` trigger must name each workflow | +| 2. A scheduled poller | One job on a timer lists every open item in every repo in a few queries and evaluates them all in one process | W-1, W-3 to W-9. Nothing at all in any Consumer | W-2: a schedule cannot be finer than five minutes, about 60% of ticks are dropped here, and gaps of hours have been observed | +| 3. A looping job | One job ticks every 30 seconds for some hours, then starts its successor | W-1 to W-8 | W-9 in spirit. One of 20 slots, permanently. GitHub's terms do not say whether this is allowed | +| **4. A poller with nudges** | Option 2 is the base. An event carries no work: it only asks for a pass now. The poller holds one concurrency group, so any number of nudges during a pass collapse into exactly one more pass | W-1, W-3 to W-9, and W-2 while the relay is up | W-2 when the relay is down, when only the schedule remains | + +**Decided: option 4, built as option 2 first.** + +- Passes cannot overlap, and at most one waits behind the one running. A flood of events costs at + most back-to-back passes: one runner slot, 5% of the pool. +- A pass ends by starting its successor if anything is due sooner than the next tick. Scheduled + ticks then only restart a broken chain. +- The Factory's own writes cannot run away. A nudge asks for a pass; whether a Run is admitted is + still Admission's decision, under the brakes. +- One pass is the only decider, so an org-wide limit is a decision inside one process. No lease and + no race. +- A Run is still its own workflow run, holding its item's lane from `run` to `apply`. +- Several staggered schedule lines on the poller restart a broken chain. A second scheduled + workflow was considered as a watchdog and dropped: it is the same unreliable scheduler twice. +- **A hosted relay supplies the low latency, and holds no credential over the Factory.** One small + stateless function outside GitHub does two things. It receives the Controller App's webhooks, + which GitHub sends for every repo the App is installed on, and it runs a timer. Each webhook and + each tick becomes a nudge. It checks each webhook's signature, keeps no state and decides + nothing. This is what replaces a nudge file in every Consumer repo: no file anywhere, and events + heard everywhere. +- A nudge carries nothing and passes coalesce, so the relay causes one through something anyone may + do: for example, starring a designated repo. There is then nothing on the relay worth stealing; + a credential able to start a workflow could also cancel runs and delete run records. Anyone can + cause a pass this way, and the cost of that is already bounded at one runner slot. +- The relay is the one permitted exception to "no always-on service". Correctness never depends on + it: if it is down, the Factory falls back to GitHub's schedule and only latency suffers. Which + host, and which public event, are settled by trial (P-20, P-31). +- The Factory's own repo nudges the poller directly, from a workflow of its own, from the first + day. +- Only the poller can be started from outside the Factory, and it takes no inputs. A Run starts + from a recorded Decision (§3.5), so a credential that can start workflows cannot direct one. +- Option 3 is held until GitHub's terms are answered. Option 1 is held unless a requirement appears + for a response in seconds; none is in the PRD. + +### 7.4 What was set aside + +| Option | Why not | +|---|---| +| A branch in the Factory repo holding State records and a Ledger | A second authoritative copy of what GitHub holds; contended under many writers | +| The same in a private data repo | The same, and the source it would hide is already public | +| A record on the item: a check run's output or one comment | Nothing an issue can carry; a comment record is a marker again | +| Events batched as artifacts and committed by one job | Two paths to keep correct, for a log Actions already keeps | +| A database | Needs an always-on service | + +### 7.5 Build the Kernel, or adopt one + +Researched on 2026-10-11 from project repos, release pages and official documentation. Nothing was +installed or run. Versions and dates are as read that day. + +**Verdict: build the Kernel, and adopt only libraries that run at test time.** No framework fits +"a pure function over facts read afresh on every pass, with nothing stored" better than a few +hundred lines of the Factory's own code. + +| Part of the Kernel | Verdict | Why | +|---|---|---| +| The Lifecycle | Build: versioned data, its own interpreter and graph check | The one credible library, XState, advances a stored snapshot by events. This Kernel works State out from facts each time. Its graph tools list paths; they do not prove that every state ends | +| Admission and the gate | Build, as plain functions | The policy engines (OPA, Cedar, Zen, CUE) are separate binaries or native code. Using one turns each decision into a call to another process, in a second language, and slows a simulator that needs thousands of runs | +| The authority table | Build, as data with a schema | Cedar's model is the nearest fit and is far larger than a table needs | +| The simulator | Build it; adopt a property-testing library to drive it | Hypothesis in Python and fast-check in TypeScript are equivalent, one dependency each, and both replay a failure from a seed | +| The formal model held in reserve | Quint, in preference to TLA+ written by hand | It has a seeded simulator of its own and hands the same specification to the model checkers | +| Schemas | Keep JSON Schema | CUE is a separate binary and is not yet at version 1 | + +Two things are worth copying and not running, since both need a server: the shape of +`palantir/policy-bot`'s rule file, where named rules are combined with "and" and "or"; and the +design of Prow's Tide, which also reads the platform's state afresh on every pass. + +Excluded because each needs a server, a database or both: Temporal, Restate, DBOS, Inngest, +Hatchet, Zuul and Kodiak. Mergify's engine is no longer open, and bors-ng is archived. The agent +frameworks, LangGraph among them, offer nothing to a deterministic function. + +**What would change the verdict.** + +| If | Then | +|---|---| +| The Lifecycle comes to need nested or parallel states | Adopt XState, which means TypeScript | +| Policy has to be edited or audited by someone other than the Maintainer, or vary by repo | Adopt OPA, in the job that decides and never in `apply` | +| The simulator finds a fault in ordering that it cannot explain | Make the Quint model a required check | + +**What this means for the language.** No candidate forces the choice. The facts that bear on it: + +| Fact | Favours | +|---|---| +| The runner's default image has Python 3.12 and Node 22, with newer versions of each cached | Neither | +| Node runs TypeScript files directly without a build, but the Node 22 documentation still calls that a release candidate; it is stable from Node 24.12 | Python, slightly, until the runner's default moves | +| Run that way, TypeScript loses enums and some other syntax, and nothing checks the types; a type checker is still a development dependency | Neither: Python needs one too | +| A schema validator with no dependencies exists for Python, though only up to draft 07; the usual TypeScript one has four | Python, slightly, because `apply` should carry as little third-party code as possible | +| The page in the control room is JavaScript whichever is chosen | TypeScript, slightly | +| Adopting XState would require TypeScript | TypeScript, only if the first row of the table above comes true | + +**Recommendation: Python, standard library only at run time, with Hypothesis as a test-only +dependency that never enters `apply`.** A laptop needs a current Python installed; the system one +on macOS is too old for the test library. Pin the runner image by version, not `ubuntu-latest`, +since the default will move. + +**The trial of both.** On 2026-10-11 the Kernel and the validator of the proof of concept (§9.13) +were ported to TypeScript with no dependencies, and run directly by Node with no build step. Every +decision the Python Kernel makes in the simulator and in its unit tests was recorded, 10,785 in +all, and replayed through the port. + +| | Python | TypeScript | +|---|---|---| +| Same decisions | The reference | All 10,785 reproduced exactly | +| Kernel size | 341 lines | 222 lines | +| Validator size | 90 lines | 61 lines | +| Time per decision | 13 microseconds | 4 microseconds | +| Start-up | 0.05 seconds | 0.2 to 0.3 seconds | +| A type error at run time | Runs anyway | Runs anyway | +| A type checker | A development dependency | A development dependency | +| Asserting that the Kernel does no I/O | From the syntax tree, with the standard library | No parser in the standard library: a text match, or a dependency | +| Runs with no build on the runner's default | Yes | On Node 22, where it is still called a release candidate | +| The simulator and the mutation check | Written | Not ported | + +Neither language is disqualified, and the differences in speed and start-up are far below anything +that matters beside a call to GitHub. What the trial changes is the cost of the choice. The +recorded decisions are a specification that either language can be held to, so the choice can be +revisited later without a rewrite on trust. The Maintainer ruled for Python on 2026-10-11. The +recommendation rested on two of the rows: +the purity check, which is the boundary this whole design rests on, is real in Python and +approximate in TypeScript without a dependency; and Python needs no qualification about the +runner's default interpreter. + +Sources: the runner image's software list +(); Node's +TypeScript documentation (); XState's pure transitions +(); OPA's command line +(); Quint (); +Hypothesis's stateful testing (); +policy-bot (); Tide +(). + +Not verified: whether a Quint run repeats exactly from its seed. The merge queue's availability to +a public repo on the free plan is not stated in the documentation; it is in use on +`.github-private`, which settles it. + +## 8. The control room + +### 8.1 Purpose + +`petry-projects/software-factory-control-room` houses the Factory's view and presents it to the +people who operate it, for understanding and for action. It is a public repo holding files the +Controller writes and a static page that reads them. + +It was shaped by a separate analysis of what a situation room for the fleet should be: +`petry-projects/.github-private#2242 — Situation-room view for the agent fleet: build it from +native Claude Code parts, not a command-center product`. That analysis surveyed fourteen products +and found that each shows only the agents it launches, so none can show a fleet that runs as +workflows. Its verdict was to build a view over GitHub. §8.3 says what is taken from it and what +this design changes. + +### 8.2 Objectives + +| # | The control room | Serves | +|---|---|---| +| CR-1 | Answers "what needs me?" first. Items are ordered by the human action they need, and each states the decision, the evidence and the choices | UJ-2, FR-27 | +| CR-2 | Shows one definition of State. Every status is computed by the Kernel, the same function that decides. The page computes nothing | FR-2 | +| CR-3 | Is never a source of truth, and the Factory never reads it. It is rebuilt from GitHub, and the page computes the view's age in the browser, so a stalled Factory shows as stale without anything having to run | FR-43 | +| CR-4 | Is read-only. A person acts in GitHub, through a declared human transition, and the page links there | FR-22 | +| CR-5 | Covers every Consumer repo on one page | PRD §1 | +| CR-6 | Shows cost and capacity: a budget gauge, cost per change, the share that was rework, cost by Role | UJ-8, UJ-9, FR-38 | +| CR-7 | Shows whether the Factory is adopted, used and better: the scorecard, each Lens's contribution, and the measures of §9.2 | FR-42, FR-69 | +| CR-8 | Links every number to the Runs behind it | UJ-9 | +| CR-9 | Replaces recurring report issues with one page | FR-44 | +| CR-10 | Is one feed with several lenses. The files are the contract; a page, an Artifact or a mod only renders them | §8.5 | +| CR-11 | Costs no model tokens to refresh. The poller writes it as a by-product of deciding | C-3 | + +It is not a control plane. It starts no Run, steers no agent and stores no agent state in a project +board. It is also not where a chat session's output is made terser; that is a session setting. + +### 8.3 What is taken from the situation-room analysis + +| The analysis says | Here | +|---|---| +| Build, do not adopt a command-center product | Adopted. The PRD's market research reached the same finding for the whole Factory | +| Never store status; compute it and show the snapshot's age | Adopted, and taken further: it is computed once, by the Kernel, so the board cannot disagree with the Factory about why something waits | +| Columns: Working, Needs you, In review, Ready to merge | Adopted as the attention classes of §8.4, with Stale added | +| Order rows by the human action required | Adopted as CR-1 | +| Feed the board from a GraphQL snapshot taken by a chat session | Replaced. The poller already reads every open item on every pass; the board reads its output. No session has to be open, no tokens are spent, and no interactive API quota is used | +| Include a budget gauge fed by per-run accounting | Adopted as CR-6. The accounting is the residue of §3.9 | +| Decisions as choices, answerable from a phone | Supported: a Hold that needs a decision carries enumerated choices, so any lens can render them as buttons | +| Decisions still execute in GitHub | Adopted as CR-4 | +| One org-level project as the pipeline view, with no custom agent-state fields | Compatible, and outside the Factory. The control room's pipeline panel shows the same thing from computed State | +| A self-written mod as a heads-up display | A lens over the same files (§8.5). It reads the view and does not poll GitHub | +| Keep it private; a public page on a public repo is world-readable | **Not adopted as written.** See §8.6 | + +### 8.4 The feed + +| File | Holds | +|---|---| +| `view/meta.json` | When the view was written, by which pass and Release, and how far back it reaches | +| `view/items.json` | Every open Work item: computed State, reason, attention class, next actor, the decision required, age, attempts, Runs so far against the budget of 10 a merge, cost, links | +| `view/runs/.jsonl` | One line per Run: the run record and the usage and outcome from its artifact | +| `view/findings.json` | Open Findings with category, severity, owner and Disposition; and per reviewer, how many were adopted and how many unadopted ones were later proved right | +| `view/scorecard.json` | The success metrics of PRD §9 and the adoption and use measures of §9.2 | +| `view/comparison.json` | The weekly comparison of the POC and the Factory (§9.2) | +| `view/capacity.json` | Usage per window, the estimate of what is left, and the degradation stage | +| `view/health.json` | Whether Admission is paused and why, when the last pass ran, and recent capacity and environment faults | +| `view/consumers.json` | Each Consumer repo's ring, and what its readiness check finds missing | + +```json +{ + "schema": "view-item/1", + "item": { "repo": "petry-projects/software-factory", "type": "pull_request", "number": 42 }, + "state": "held", + "reason": "max_rounds_reached", + "attention": "needs_you", + "next_actor": "maintainer", + "decision_required": { + "question": "rounds_exhausted", + "choices": ["grant_another_round", "accept_as_is", "close"], + "evidence": ["finding:f_22ab", "run:r_01JA7Q"] + }, + "content_id": "sha256:5d0c", + "age_hours": 19.5, + "attempts": { "fix-review-findings": 3, "adjudicate-findings": 3 }, + "runs": 9, + "cost_usd": 4.8, + "sources": ["run:r_01JA6Z", "run:r_01JA7Q"], + "url": "https://github.com/petry-projects/software-factory/pull/42" +} +``` + +The page reads these files directly and is rebuilt only when its own code changes. + +The feed carries no free text from outside. An item is a repo and a number, and reasons are codes. +A title is a separate field marked untrusted. The page shows it as text. A lens that feeds a model, +such as a board inside a Claude session, drops it: that session holds the Maintainer's own +permissions, and a pull-request title is something a stranger writes. + +`question` and `choices` are enumerated codes. The sentence a person reads is rendered from them. + +Attention classes, in the order the page lists them: + +| Class | Meaning | From | +|---|---|---| +| `needs_you` | A decision, an approval or an answer is required of the Maintainer | A Hold that needs a decision; an open `human` Finding; a `security` Finding awaiting Override; an `outside` item awaiting admission | +| `failed` | Verification failed and no Run is addressing it | `addressing` with attempts exhausted, or a required check red | +| `review` | Waiting on Lenses or owners | `verifying`, `adjudicating` | +| `ready` | Converged and approved, or in the queue | `converged`, `queued` | +| `working` | A Run holds the item | A run record in progress | +| `stale` | Unevaluated past its state's limit while passes were running | I-1 breached | +| `not_served` | Outside what the Factory serves | A pull request that does not target a protected branch | + +The class is computed by the Kernel beside the State, so the status on the item and the row on the +page are always the same answer. + +### 8.5 Lenses + +| Lens | Reads | For | +|---|---|---| +| The status on the Work item | The Kernel's projection | One item, where the work is | +| The dashboard page | The feed | The whole portfolio, on any device | +| A private Artifact board in the Maintainer's Claude Code session | The feed | The decision queue, beside the session that acts on it | +| A mod band in the desktop Code tab | The feed | A heads-up count of what needs a person | + +### 8.6 Public or private + +The Maintainer chose a public view with a public page. The situation-room analysis says to keep the +board private. They are reconciled as follows. + +- **The view hides nothing by being private.** Its sources are the issues, pull requests, run + records and artifacts of public repos. The weekly cost report is already a public issue. +- **A free plan publishes a page only from a public repo.** +- **What a public page does add is aggregation.** One page that lists every held item and how much + budget is left makes the Factory's state easier to read for anyone, not only its operator. +- **The private lens remains.** The Artifact board of §8.5 is private by construction and reads the + same files. + +Decided on 2026-10-10: every panel is public, the budget gauge included. + +## 9. Build order and stopping + +### 9.1 Order + +The review Intent comes first, so that the Factory reviews itself and everything built after it. +The first slice is defined by one event: **the Factory approves one of its own pull requests.** +Anything not needed for that event is in a later slice. + +| Slice | Delivers | Who approves Factory changes | +|---|---|---| +| 0. Foundations | A small proof of concept of the Kernel, the schemas and the validators, built first (§9.13). The context the Factory's authors work from: `AGENTS.md`, standards, a decision log and the lessons of the POC (§9.14). The repo, its ruleset and `factory / verify`. The schemas the first slice uses, with golden examples. The pull-request Lifecycle. The rubric for `pr-review`. The simulator with a skeleton Kernel. Three Apps: the Controller's, the reviewer's and the shared reader. The trials of §6 that the first slice rests on, each leaving a fixture | The Maintainer | +| 1. First approval | The facts and records ports. The Kernel's Lifecycle, Convergence and gate. The poller, nudged directly by the Factory's own repo. `run` and `apply` for the review Intent, with its eval set. Deterministic checks read as Findings. The status on the item. A feed of two files, `meta` and `items` | The Maintainer, until the first Release on `next` serves the Factory's repo; then the Factory's reviewer, with the Maintainer able to block | +| 2. The Factory's own author | The fix and implement Intents under the run contract, with `verify`. Acceptance contracts drafted by `qa-lead` and approved by the Maintainer (§4.8). From here the Factory writes most of its own code | The Factory | +| 3. A Consumer repo, and the weekly comparison | The first `stable` Release, promoted on evidence from `next`. One low-risk Consumer, cut over by pause and repoint (§9.6). The relay. The weekly comparison of the POC and the Factory, as a file and one tracking issue (§9.2) | The Factory | +| 4. The control room | The rest of the feed and the dashboard page. Adoption counted per Advisory reviewer. Runs per merge on every item | The Factory | +| 5. Domain Roles in shadow | The reviewing and adjudicating Intents of `qa-lead`, `security-lead`, `solution-architect`, `devops-lead` and `sre-lead`, each with its rubric and eval set, running on the reader App on a sample and binding nothing. Promotion follows, one Role at a time, and brings that Role its own App | The Factory | +| 6. Onward | The initiative driver and the remaining personas, as in the review's migration shape | The Factory | + +The control room and the shadow Roles come after the Factory's own author, on the Maintainer's +decision of 2026-10-11. Until slice 2 every line of Factory code is written in the Maintainer's own +sessions (§9.12), so the sooner the Factory can write, the sooner that load falls. Until slice 4 +the Maintainer has the status on each item and the two feed files. + +**Slice 0 is done when** the proof of concept has answered its questions; an agent given only the +repo's own context can add a rule to the Kernel and pass the required check without being told +how; `factory / verify` is a required check and passes; every schema the first +slice uses has an example that validates; the Lifecycle passes its exhaustive checks; the simulator +runs its fixed seeds against a skeleton Kernel; the three Apps exist; and each platform trial the +first slice rests on has a recorded fixture. + +**Slice 1 is done when** a Release on `next` has reviewed a pull request to the Factory's repo, the +machine user's approval of it stands, the pull request has merged through the queue, and its status +block said why at each step. + +**The critical path to that event,** worked backwards from it: + +1. The pull request merges through the queue: the Factory repo has a ruleset, a merge queue and + `factory / verify` running on `merge_group`. +2. The machine user's approval stands: it is a code owner for the paths the pull request touches, + and it is neither the author nor the last pusher. The Maintainer's own sessions therefore push + as the Maintainer. +3. `apply` submitted that approval: the effects and identity ports exist, the reviewer's App and + the machine user's token are environment secrets, and the Result validated. +4. `run` produced the Result: the Harness built a Task from a Decision, the engine port ran the + pinned CLI with the tools that Intent is allowed, and the Factory's model credential exists. +5. The review Intent has a prompt, a result schema and **an eval set**. A gate-deciding Intent may + not serve without one (FR-55), so the eval set is on the critical path. Its cheapest source is + the POC's own history: merged pull requests later reverted or fixed are known answers. +6. The poller started the Run: the Controller's App exists, and a pass recorded the Decision. +7. The Kernel reached that Decision: the pull-request Lifecycle, Convergence and the gate, with the + deterministic Lenses already reported. Those Lenses are workflows in the Factory repo: lint, + workflow security, code scanning. +8. The pass read the facts and the run records: the two ports, with completeness flags. +9. All of it ran from a Release on `next`: something cut that Release and moved the channel + (P-30), and the entry shim called it. +10. Every pull request that built the above was approved by the Maintainer, under the count of + §9.2. +11. Before any of it, the first batch of trials (§6.1). + +**Not in slice 1:** the fix and implement Intents, acceptance contracts, adoption measurement, +domain Roles, path scopes, the relay, the dashboard page, the private +Artifact board, the comparator against the POC, any App beyond the three named, any Consumer repo. + +Until the first Release on `next` serves the Factory's repo, the Maintainer's attention is the +line's capacity. That is why slice 1 is cut this narrowly, and why §9.2 counts the approvals it +takes. No change reviews itself: the Release that reviews a pull request was cut before that pull +request existed. + +One rule protects recovery. The Maintainer's own approval and the hermetic required check never +depend on the Factory running. When the Release that serves the Factory is broken, that is the way +out. + +Using the POC's reviewer for the bootstrap was considered. Its comment gate can be satisfied only +by the POC's author Role and has no switch, so it would need a change to the POC first. It remains +the fallback if the count in §9.2 is passed. + +### 9.2 Stopping + +Decided by the Maintainer on 2026-10-10: there is no period and no automatic rule. If the Factory +is not adopted and used, the Maintainer decides to stop. This replaces both the parity rule and the +outcome-based rule recorded earlier that day. + +The control room exists to make that an informed decision. It shows: + +| Shown | Why | +|---|---| +| Consumer repos served, and the share of the org's merged pull requests the Factory handled | Adoption | +| Runs a week, and how often the Maintainer overrides or works around the Factory | Use | +| Standing approvals, issue-to-merge time and runs per merge, against the POC's baseline (at most 40%, 24.5 hours, about 1,500) | Whether it is better | +| Escaped defects | Whether "better" was bought by weaker review | + +None of these stops anything by itself. + +**A weekly comparison.** From slice 3 a scorecard is produced every week, automatically, setting +the POC beside the Factory on the same measures: standing approvals, time from issue to merge, runs +per merge, and escaped defects. The poller already reads every repo, served or not, so the POC's +side costs nothing extra to measure. It is a file in the feed and one tracking issue that is +updated, never one issue a week. This replaces a single re-measurement on a fixed date: the +Maintainer sees every week whether the Factory is ahead. + +With no period set, the decision needs a moment at which it is made. Two events serve: + +| Decision point | The Maintainer records | Shown beforehand | +|---|---|---| +| The first Release on `next` that serves the Factory's repo | Continue or stop | How many Factory pull requests the Maintainer has had to read and merge, against a starting figure of 40 | +| The first Consumer repo served | Continue or stop | The adoption and use measures above | + +The control room also shows what share of Factory pull requests needed the Maintainer only because +they touched a protected path (§5.1). Early on that will be most of them, and the approval count +should be read with it. + +The record is a file in the Factory repo. Passing 40 approvals before that Release stops +nothing; it is the signal to shrink the slice further or to fall back on the POC's reviewer. + +### 9.3 What a pre-mortem changed + +A pre-mortem on 2026-10-10 assumed the Factory had been abandoned and asked why. Seven causes were +ranked, and each changed this document. + +| Likely cause of failure | Change made | +|---|---| +| The bootstrap starves on the Maintainer's attention | Slice 1 cut to one event, with a written list of what is out (§9.1); approvals counted (§9.2) | +| Domain owners make it slower and no better | Owners start in shadow and are promoted on evidence (§5.2) | +| Third-party threads become the new comment gate | Only required Lenses gate; other Findings gate only when the reviewer adopts them (§4.2) | +| The simulator passes and GitHub disagrees | The fake GitHub is grounded in recorded responses (§4.5, §6) | +| The poller goes quiet | Staggered schedule lines, an optional external timer, and staleness computed in the browser (§7.3, §8.2) | +| Two systems run forever | Decision points keyed to events (§9.2) | +| The model window runs out | Shadow Roles run on a sample; Runs per merge shown on every item (§3.12, §8.4) | + +### 9.4 What a simplification pass removed + +A second pass on 2026-10-10 asked of each part what requirement would fail without it. + +| Part | Replaced by | +|---|---| +| The control room as an input to decisions | Decisions read GitHub only. The view is written and never read back (§3.9) | +| A separate intake Role typing third-party prose | Adoption by `pr-review`, whose own Finding carries the fields (§3.7) | +| Dispositions as standing records | The thread, resolved by the owner's App; the two halves travel in Results (§3.8) | +| An App for every Role from the first day | One shared reader App; a Role's own App when it first writes (§5.1) | +| A watchdog workflow | Staggered schedule lines, and an optional external timer (§7.3) | +| The Wake contract and a next-evaluation time on every Decision | A nudge that carries nothing; a staleness limit per state (§3.2, §4.1) | + +Kept after the same question: the three-job split, content identity, the four brakes, the +simulator, a separate control-room repo, and both channels with their rings. Both channels and the +ring file exist from the first commit, because the entry shim is frozen and should have its final +shape from the start. The Factory's own repo is served by `next` (§9.6); `stable` is unused until +there is a Consumer. + +### 9.5 What a failure-mode pass added + +A third pass on 2026-10-10 went component by component: how each fails, how anyone would know, and +what it takes down. + +| Component | Failure | Change made | +|---|---|---| +| Facts port | A partial read taken as "nothing open", giving a false approval | Completeness flags; incomplete is unreadable (§3.3, I-20) | +| `apply` | Writing on an item held or changed while the Run waited | `apply` re-reads the item first (§3.6, I-21) | +| Run start | Anything able to start a workflow directing a Run | A Run starts only from a recorded Decision (§3.5, I-22) | +| Whole Factory | One outside cause counted as an attempt on every item | Faults classed; environment faults pause Admission once (§3.10, I-23) | +| Engine | The model window running out mid-Run | A `capacity` outcome defers model Runs and is the capacity signal (§3.10) | +| Kernel | A wrong decision nobody can reproduce | Every pass records what it read and decided (§3.4) | +| Required check | A random seed blocking every merge | A fixed seed list gates; exploration is scheduled (§4.5) | +| `verify` | A forged or unrepresentative pass read as the verdict | `verify` is a filter; the Consumer's checks are the verdict (§2.3) | +| Control room | The page lagging the feed | The page reads the feed directly (§8.4, P-21) | + +Found sound as designed: a partial `apply` heals at the next pass, because State is derived from +what was actually written; a lost nudge, a dropped tick or a dead timer costs latency only; and a +wrong control room changes no decision. + +### 9.6 Which channel serves the Factory's own repo, and how a Consumer is cut over + +A consistency pass found that the first `stable` Release could not be reached. ADR-0013 promoted a +Release to `stable` only on verified Effects from the `next` ring, the Factory's own repo was +always served by `stable`, and nothing was on `next`. Two ways out were compared. + +| | A canary repo on `next`, the Factory's repo on `stable` | The Factory's repo on `next` | +|---|---|---| +| Evidence for promotion | Synthetic work items someone scripted. They test what was thought of, which the simulator already does | Every real pull request to the Factory: varied, numerous, and unscripted | +| What gates the Factory's own changes | A Release promoted on that thin evidence | A Release that is cut and tagged but not yet promoted | +| A bad Release | Reaches the Factory's repo only after promotion | Reaches the Factory's repo at once, and only that repo | +| Recovery | Move `stable` back | Move `next` back; the Maintainer's approval and the required check do not depend on it | +| Speed of the loop | A fix to the reviewer waits for promotion | A fix serves the Factory at the next cut | +| Extra parts | A repo and a library of synthetic items to maintain | None | + +**Decided: the Factory's repo is served by `next`.** The canary's apparent safety is thin, because +the Release it would protect the Factory from is promoted on evidence weaker than the Factory's own +traffic. Serving the Factory from `next` is ordinary practice: the team that builds a thing runs it +first. What still protects the Factory's own merges is the deterministic required check, the brakes +outside the Kernel, and the Maintainer's ability to block. + +The self-hosting rule is restated to match. No change reviews itself, because a Release is cut from +what is already merged. A Release reaches `stable`, and so any Consumer, only on evidence from +`next`. + +**Cutover is pause, then repoint.** With a poller there is nothing in the Factory for a Consumer's +workflow to call, so cutover is two switches in two repos: + +1. The POC's channel tag for that Intent family moves to a commit where the family does nothing. +2. The Factory's ring file lists the repo for that family, and takes effect at its next Release. + +They cannot change together. In this order the gap is "nobody serves that family on that repo for a +while", and never "both do". The Maintainer accepted the gap on 2026-10-10. Rollback is the same +two switches reversed. + +### 9.7 What a consistency pass found + +An independent reader compared the five documents on 2026-10-10 after about ten rounds of edits. +Every cross-reference resolved. Three findings were design holes and are settled above and in +§7.2: the unreachable first `stable` Release, a spend cap that survived after money stopped gating, +and a cutover that still called a mechanism the poller had removed. The rest were statements left +behind by earlier decisions, and were corrected in the same change. + +### 9.8 What the last two passes changed + +A pass that read the design as each party to it, and a pass that followed each decision one step +further, both on 2026-10-10. + +| Finding | Change made | +|---|---| +| Nothing pushed to the Maintainer, and the controls were never listed | The control vocabulary; assignment and review requests as the notification (§4.7) | +| A plain comment cannot be resolved, and anyone can open a thread on a public repo | A plain comment steers; only a review from a person with write access gates (§4.2, §4.6) | +| A stranger's comment would reset an agent's attempts | Only a person with write access resets them (I-28) | +| Failing acceptance tests would turn the default branch red | One pull request, with the contract as its first commit (§4.8) | +| The contract's drafter and the implementer share blind spots | A different model family for the drafting Intent where configuration allows (§4.8) | +| `pr-review` becomes the single funnel for outside views | Unadopted Findings later proved right are counted (§4.2) | +| The Maintainer's notifications flood | One notice per item per reason; backpressure from the queue (§4.7, I-32) | +| A pass's cost grows with the backlog | Issues come under control by a label (§4.7) | +| A held-out set readable by the Role it tests | Held-out sets under a different owner (§3.13) | +| A Role in shadow forever | A shadow budget (§5.2) | + +### 9.9 What is not yet designed + +Found by explaining the design on one page and by comparing it with the PRD group by group. + +| Requirement | State | +|---|---| +| FR-1, for issues | The issue Lifecycle is a list of state names. How an issue becomes a pull request is not designed | +| FR-6, resumable Runs | Not designed. A Run that times out pushes nothing and starts again | +| FR-12, the injection test suite | Required before promotion; not designed | +| FR-14, shared-branch safety | The rule is stated; the push guard is to be carried over from the POC | +| FR-21, FR-36, priority and graceful degradation | Backpressure and the capacity outcome exist. The stages between full service and Pause are not designed | +| FR-38, complete cost attribution | Met for every Run that ends. A killed Run's cost is an estimate from its duration | +| FR-42, the scorecard | The feed file is named. How each metric is computed is not designed | +| FR-45 to FR-49, release | Channels, rings and rollback are decided. The tooling that cuts and promotes is not designed | +| FR-55, evals | Required for every gate-deciding and code-writing Intent. None is built | + +### 9.10 What the day's changes bought + +The POC, ADR-0013 as first drafted, and this design, against the things that cost effort to build +and to run. + +| | POC | ADR-0013, first draft | This design | +|---|---|---|---| +| Authoritative stores beside GitHub | Labels and about eighty kinds of marker | A git-backed State record and Ledger | None | +| How work is noticed | An agent per event; fifteen workflows to recover what was lost | A wake per event, with leases and a sweep | One poller; an event only asks for a pass | +| Files a Consumer repo must carry | Several stubs per agent | One | None required | +| What gates an approval | Every comment from nine reviewers | Structured Findings from every Lens | Required Lenses, people with write access, and acceptance evidence | +| What a change is judged against | Not defined | Not defined | An acceptance contract written first | +| Who decides a Finding | The author's own claim, verified afterwards | One gate function | The owner of its category; domain Roles in shadow until earned | +| Where the Maintainer looks | Each item, and machine-filed issues | A status block per item | A status per item and one public page | +| History | Unbounded but unqueryable | 180 days | 90 days | +| Parts specified before building | None | Twelve decisions | Contracts with examples, 36 invariants, a simulator, 30 trials | + +What it cost: history is half as long; money no longer gates, only a count of Runs does; latency +depends on an outside tick; there are four repos and, in time, about ten Apps; and nine PRD +requirements were reworded to match (§10). + +### 9.11 The case for not building this + +The strongest case for the option the review rejected, hardening the POC, is made here so that it +is answered and not ignored. + +- Several of this design's rules are policy, and the POC could adopt them without a rewrite: only + required Lenses gate, deterministic checks before a model review, no branch updates without a + conflict, attempts counted from run records. The review said the first of these would do more + for the largest failure class than the rewrite, and could be tried in weeks. +- The same control plane delivered 79 merges a week at a median under an hour in July. What changed + was the number of gates an approval had to survive. +- The Maintainer's attention is the binding limit, and a rebuild spends it: bootstrap approvals, + two systems, acceptance contracts, and a design that grew by eight review passes in a day. +- This repo's record on finishing migrations is poor. + +What that case cannot reach is structural. The POC's model holds the write credential, nothing +admits Runs in one place, there is no single Release, and State is implicit. Four of the review's +seven root causes sit there, and no policy change moves them. + +The case does land one point, which is put to the Maintainer: the configuration fixes and the +narrower gate were recommended as worth having whatever else is decided, and as the cheapest test +of how much of the rest is needed. They are now tracked outside this document, in +`petry-projects/.github-private#2276 — Epic: fixes to the current factory that are worth making whatever is rebuilt (lessons from the factory MVP review)`. + +### 9.12 How it is built before it builds itself + +Until slice 2 the Factory has no author of its own. Its code is written in the Maintainer's +interactive sessions: Claude Code in the desktop app, the strongest available model, and the BMAD +workflows vendored in `.github-private`. The POC's author agent does not serve the new repo, since +that would bring the owner account's token and the POC's rules into a repo meant to start clean. + +**Whose name the work carries.** In those sessions three identities are set separately, and all +three must be the Maintainer's: the account that opens the pull request, the account that pushes, +and the commit author. A session inherits the first two from the machine's GitHub login. It does +not inherit the third, which comes from git's own configuration and has to be set in each clone. +The machine user never authors, pushes or opens a pull request there. + +**No one can approve their own pull request.** While the Maintainer is the author of every pull +request, GitHub will not accept the Maintainer as its approver. So through slices 0 and 1 the +ruleset requires the verification check and the merge queue and no approving review: the Maintainer +reads and merges. Required review, with the machine user as code owner, is switched on when the +Factory's reviewer serves the repo. The rule that the Maintainer alone owns the Factory's rule +files (§5.1) takes effect when an App, and not the Maintainer, authors the changes. Wherever this +document says the Maintainer approves during the bootstrap, read "reads and merges". + +What makes that way of working produce good code: + +| Practice | How | +|---|---| +| One story to a session | Each session starts from a written handoff and a plan, and ends at one pull request with at most four acceptance criteria. Long sessions lose their thread | +| Stories before code | Epics and stories, then a readiness check of PRD, architecture and stories against each other, then one story file per piece of work | +| Failing tests first | The test-architecture workflow writes the acceptance tests before the implementing session starts, as `qa-lead` will later do by itself | +| The specification runs | Schemas, invariants and the simulator are in the repo. One command verifies locally exactly what the required check verifies | +| Hooks hold the line | After each edit, the fast checks run. A session cannot end with the verification failing. A push as the machine user is refused. An edit to a protected path is refused unless the Maintainer asked for it | +| A reader that did not write it | Before a pull request, a fresh context reviews the diff, and a separate pass looks only for security faults. For a change to the Kernel, a second model family reads the diff once and reports. Reviewers report; they do not converse | +| Tests that can fail | Mutation testing on the Kernel, which is pure functions and so cheap to mutate | +| Types as the code is written | A type checker and a linter are required checks, and a language-server plugin shows their errors to the agent as it edits | +| Trials leave fixtures | Every platform trial records the real response, which the simulator's fake GitHub is then held to | +| The Maintainer's attention | Replies lead with the verdict, decisions arrive as choices, and status lives on a page, as set out in the situation-room analysis | + +On plugins: the built-in review, security-review and orchestration features are preferred to +community plugins that install hooks or run with broad reach. One such plugin is close in spirit to +this design, a council of six specialist reviewers; it should be read before it is installed, since +a plugin's hooks run with the session's own permissions. + +### 9.13 The first proof of concept + +Before the repo, the Apps or any credential exists, a small proof of concept is built: the Kernel, +the schemas and the validators, and nothing that touches GitHub. It needs no decision from the +platform trials, and it tests the claim this whole design rests on, that the decisions can be a +pure function which is checked before it is trusted. + +| It contains | It answers | +|---|---| +| A schema and a golden example for the facts, the Decision, the Finding, the acceptance contract, the rubric and the authority table | Can every contract of §3 be written down and checked mechanically? | +| A validator for those schemas with no third-party code | Is the standard library enough for what `apply` must check? | +| The pull-request Lifecycle as data, and its exhaustive check | Does the Lifecycle of §4.1 terminate, and is every cycle bounded? | +| The Kernel's decision for a pull request: derived State, Convergence, Admission and the attention class | Can State really be derived from facts and run records alone? | +| The boundary cases of §4.6 as tests, and a first seeded simulator with two faults | Do the invariants hold, and does a failure replay from its seed? | +| A check that the Kernel imports nothing able to do I/O | Can the boundary be asserted by a tool? | + +It is written in Python with the standard library only, as recommended in §7.5, and is where that +recommendation is tested. It is built locally and becomes the first commits of the Factory repo. +It is not a throwaway: what survives review stays. + +**What it found**, built on 2026-10-11. It is about 1,300 lines, of which the Kernel is 340, and it +needs nothing installed. + +| Question | Answer | +|---|---| +| Can the contracts be written down and checked? | Yes. Nine schemas, each with an example, and four counter-examples that must be rejected, including facts that carry prose and a Decision whose reason is a sentence | +| Is the standard library enough for validation? | Yes, in about a hundred lines, for the subset of JSON Schema these contracts use. The validator refuses a schema that uses anything else | +| Does the Lifecycle terminate? | Yes. The check also catches an unbounded cycle and a dead end when either is put in on purpose | +| Can State be derived from facts and run records alone? | Yes, with two corrections to this design, below | +| Do the invariants hold under faults? | On 300 fixed seeds, with stale reads, incomplete reads, dropped ticks, a Run missing from the list and Runs killed mid-flight. A failure prints its seed and a trace | +| Can the tests fail? | Nineteen deliberate breaks of the Kernel, one rule each; all nineteen are caught | +| Can the Kernel's purity be asserted by a tool? | Yes, from its syntax tree | +| Does it run on a stock laptop? | Yes, on Python 3.9 and 3.12 with the same results from the same seeds | + +**Two errors in this design that the simulator found.** Both are corrected above. + +- *Whose turn it is.* This document said the turn is read from the Role of the last completed Run + on the current content. But the author's answer to a thread is usually new content. On that new + content the last Run is the reviewer's re-review, so the thread was never judged and the author + was sent to fix it again, until the Run ceiling stopped it. Four in ten simulated pull requests + ended that way. With the rule corrected, 279 of 300 merge under faults. +- *A Run that has been admitted but has not started.* A pass that cannot see it admits another. + The records port must report a Run that is waiting for a runner, and the Kernel must count it + as holding the item. + +**One rule the simulator added** (I-38). When the list of runs lags behind a Run just started, a +pass can still admit a duplicate. A Run therefore asks the Kernel again as it starts, and does +nothing if it is no longer the Run that is wanted. + +**Smaller findings.** Seeds reproduce across Python versions only if the simulator avoids the +library's `choice`, whose sequence differs between versions. Data files are JSON, not YAML. In the +simulation a pull request takes a median of three model Runs to merge and ten at the ninetieth +percentile, against the PRD's ceiling of ten; that is a property of the made-up rates at which +reviews find faults and checks fail, and says nothing yet about the real ones. + +### 9.14 Context engineering + +Most of the Factory is written by agents, and an agent builds from what it is given to read. That +context is treated as part of the product: versioned, reviewed, kept small, and tested. Slice 0 +establishes it, on the Maintainer's instruction of 2026-10-11. + +| What | Its job | Rule that keeps it good | +|---|---|---| +| `AGENTS.md` at the root | What the repo is, the rules that are never broken, the one command that verifies, and a map of where things live | Short enough to be read in full on every task. Every rule in it names the check that enforces it; a rule no check enforces is a wish | +| An `AGENTS.md` in each major directory | The rules of that place only: in `kernel/`, no I/O and the clock as an argument | Read only when working there, so the root file stays short | +| Standards | How code, tests, workflows, security boundaries and reviews are done here | One page each. Examples before prose. The review standard is the rubric of §3.14, not a second copy | +| A decision log | Why each structural choice was made, and the boundary a check can assert | It starts again in the new repo at number one, with the current state of ADR-0013 and no history of how it was reached | +| Lessons | What the POC learned the hard way: each failure, the rule it produced, the check that holds the rule | Drawn from this repo's 1,100-line `AGENTS.md`, its postmortems, the review's root causes and defects, and the standards repo. A lesson with no check is rewritten until it has one, or dropped | +| The glossary | The PRD's terms, used exactly | One term, one meaning; a synonym is a defect | +| Skills | Procedures done rarely and exactly: run a platform trial, cut a Release, add a Role | Loaded when the task calls for them, so they cost nothing otherwise | +| Hooks and permissions | The rules that must hold whatever the agent was told | Enforcement by tool, not by instruction (§9.12) | + +Three practices come from outside this org and are adopted as they stand: keep what is always +loaded small and disclose the rest on demand; prefer a worked example to a paragraph of +instruction; and treat context like code, so that a change to it is reviewed and, where an eval +exists, measured. + +The test of the context is in slice 0's exit: an agent given only the repo can add a rule to the +Kernel and pass the required check without being told how. + +## 10. What this design changes elsewhere + +Recorded in the same change as this document. + +| Document | Change | +|---|---| +| ADR-0013, decision 2 | State is derived and never stored. The record of a Run is its Actions run record and its artifact. The control room is a rebuildable view that no decision reads | +| ADR-0013, decision 3 | The Controller is a poller. An event may only ask for a pass. The Consumer file is optional. A Run starts only from a recorded Decision. An automatic pause is derived from recent faults | +| ADR-0013, decision 4 | The Result artifact is the intent record. `apply` re-reads the item before any Effect. `verify` is a filter | +| ADR-0013, decision 5 | One App per Role, registered when the Role first writes; one shared reader App. The machine user never pushes. The Maintainer alone owns the Factory's own rules | +| ADR-0013, decision 7 | An owning Role resolving a thread is a declared Role transition, beside the human one. An owning Role starts in shadow. The reviewer resolves third-party threads it does not adopt. A plain comment steers; a review from a person with write access gates | +| ADR-0013, decision 6 | The Factory's own repo is served by `next` | +| ADR-0013, decision 10 | The review Intent on the Factory's own repo is the first component. The comparison with the POC is a measure, not a gate. Stopping is the Maintainer's decision, on adoption and use. Cutover is pause, then repoint | +| ADR-0013, decision 11 | No longer two provisional mechanisms. The poller's latency and cost are measured against the same thresholds | +| ADR-0013, decision 12 | The starting required set is named. A third-party Finding gates only when the reviewer adopts it, and adoption is what gives it typed fields | +| PRD, FR-40 and the glossary | History is 90 days. The State record is computed; the Ledger is the run records and artifacts, indexed in the control room | +| PRD, OQ-2, OQ-3, OQ-8, OQ-9 | Recorded as decided, with what remains open | +| PRD, NFR-11 | A new Role is one pull request plus one App registration | +| PRD, FR-39, FR-73, C-4 | The ceiling that gates is Runs per Work item, with the amount shown. The monthly cap applies on a metered credential. An optional external timer is permitted | +| PRD, FR-59 and the non-goals | The acceptance contract; defining what a change is judged against is in scope | +| ADR-0013, decision 12 and Consequences | Convergence requires evidence for each acceptance criterion | +| PRD, FR-2, FR-32, FR-33, FR-50, FR-51 and the glossary | Brought into line: no next-evaluation time, required Lenses in place of awaited reviewers, Dispositions read from the surface, the Factory served by `next`, the Consumer file optional | diff --git a/docs/prd/agentic-software-factory.md b/docs/prd/agentic-software-factory.md new file mode 100644 index 00000000..63fa0aa6 --- /dev/null +++ b/docs/prd/agentic-software-factory.md @@ -0,0 +1,947 @@ +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# PRD: Agentic Software Factory — Production MVP + +**Status:** Draft — for maintainer review +**Date:** 2026-10-10 +**Evidence pinned at:** commit `aef61164` + +## 0. Document purpose + +This document states what the `petry-projects` agentic software factory must do to be called +production-grade. It is written for the maintainer, who decides scope and targets, and for whoever +designs and builds against it, human or agent. + +It is one of five documents produced by the October 2026 architecture review: + +- **This PRD** — the requirements. It says *what* and *how well*, not *how*. +- [`docs/architecture/factory-mvp-architecture-review.md`](../architecture/factory-mvp-architecture-review.md) + — the review of the current system and the analysis of architectural options against these + requirements. +- [`docs/architecture/adr/0013-factory-mvp-reference-architecture.md`](../architecture/adr/0013-factory-mvp-reference-architecture.md) + — the proposed decision. +- [`docs/architecture/software-factory-design.md`](../architecture/software-factory-design.md) + — the interaction and contract design that follows from the decision. +- [`docs/research/market-agentic-software-factories-research-2026-10-10.md`](../research/market-agentic-software-factories-research-2026-10-10.md) + — market research: what the market offers, who the customer is, and where this Factory sits. + This PRD was revised against it; §7 of that document lists what changed. + +The current system is treated as a proof of concept (the **POC**). It established which functions +are possible and where the hard problems are. This PRD defines the **MVP**: the first version +suitable for production use, meaning it can run unattended across the org's repos without a human +keeping the line moving. + +Conventions: + +- Terms in §4 (Glossary) are used exactly. A synonym anywhere in this document is a defect. +- Requirements are numbered globally (`FR-n`, `NFR-n`) and are written to be testable. +- Inferences that the maintainer has not confirmed are tagged `[ASSUMPTION]` and indexed in §12. +- Measurements are dated 2026-10-10 unless stated, against commit `aef61164`. Their sources and + limits are in the review document, §2. +- Issues are cited as `repo#number — title`. `.github-private` is public; nothing here is secret. + +## 1. Vision + +A single maintainer states intent as an issue, or approves an idea, and receives a reviewed, tested, +merged change. Between those two points a set of agent roles plans, implements, reviews and ships, +across every repo in the org, including the repo that hosts the factory itself. + +The POC proved each of those functions works in isolation. What it did not deliver is the *line*: +predictable flow, bounded cost, and outcomes that can be trusted without re-reading the diff. The +MVP keeps the functions and adds six properties: + +1. **Every work item has one visible state and a known next actor.** Nobody has to read three + scripts to learn why a pull request is not approved. +2. **Nothing is lost.** A dropped event, an evicted run or a timeout delays work; it never deletes it. +3. **No claim is trusted without evidence.** "Fixed", "approved" and "done" are facts read from the + repository, not statements made by an agent. +4. **Cost is bounded before it is spent.** The factory knows what it can afford before it starts a + run, and one merge cannot trigger work proportional to the number of open pull requests. +5. **A human decision binds everywhere.** A hold stops every role; a release resumes them; the + maintainer's account is never mistaken for an agent's. +6. **A change to the factory reaches production by one gated path**, and the maintainer can say when + it will be live and roll it back in one action. + +**What is distinctive.** By late 2026 the market had made the workers a commodity: turning an +issue into a pull request, repairing CI on that pull request and giving it a first review are each +offered by eight or more products, several free on public repos ([market research](../research/market-agentic-software-factories-research-2026-10-10.md), §4.3). +What no product offers a solo operator is what surrounds the workers: + +- a merge decision taken unattended on the operator's own policy, by an approver that is not the + writer. Four vendors refuse in writing to merge to a default branch; +- a view across a portfolio of repos of why each Work item is waiting; +- one versioned definition of the automation, rolled out in stages across a dozen repos; +- work admitted against the capacity actually left, with a hard cap on spend; +- verification by Lenses from different vendors and of different kinds. + +That is the Factory's scope. It builds the line. It adopts the workers. + +**The working hypothesis is convergence.** A change is better when a council of independent Lenses +(models, tools and perspectives, each looking for something different) has examined it and their +Findings have been resolved, than when one agent has done its best alone. That holds across the +whole lifecycle: research, analysis, architecture, design, implementation, validation, release and +operation. The research supports this conditionally (review document, §15): independent evaluation +beats self-evaluation, and Lenses that differ in what they can see find different defects; but a +council that converses does worse than one that reports, and a council that co-writes does worse +than one author. The requirements in §6.11 are written to those conditions. The POC ran a large +council through free-text discussion and could not tell which Lens earned its place; the MVP runs +it through a contract and measures each one. + +## 2. Why now + +The POC grew by emergent design: each problem was met where it surfaced. That was the right way to +discover the problem space. The telemetry now shows the approach has stopped converging. + +| Signal | Measured | Source | +|---|---|---| +| Work per outcome | At least 44,968 workflow runs in 7 days in the hosting repo for 30 merged pull requests: about 1,500 runs per merge. 24% of runs were cancelled and 31% skipped. | Actions runs API, 2026-10-03 → 10-09 | +| Flow | Merges per week flat at 14–36 since late August. Median open-to-merge rose from about 1 h to 12–22 h. Commits per merged pull request rose from 2–4 to 7–10. | Pull-request history, 400 PRs | +| Review gate | 0 of 52 open pull requests org-wide hold an approval. 34% of the last 150 merged pull requests had no standing approval at merge; 20 of 29 agent-authored merges in the latest week. | GraphQL search and timeline | +| Defect inflow | Bug-labelled issues rose from 2–9 a week (July, August) to 21–36 a week (since 2026-09-07). | Issue history, 600 issues | +| Lost work | pr-review runs 37% cancelled (1,000 runs in a day for 14 open pull requests); dev-lead 42% cancelled before doing any work. GitHub drops about 60% of scheduled ticks. | `petry-projects/.github-private#2193 — PR Review Agent — failures detected 2026-10-10`; `#1741 — [bug] 42% of dev-lead runs are cancelled before doing any work`; `#2160 — GitHub drops about 60% of scheduled ticks here` | +| Capacity | The GitHub API quota shared by the fleet and the maintainer ran out on 2026-10-10. The LLM budget breaker has never fired. | `petry-projects/.github-private#2197 — The fleet and every session share one GitHub API quota`; `#1565 — Auto-pause companion` | +| Compensation | 15 workflows exist to retry, sweep, repair or audit, against 28 that do primary work. In the dev-lead and pr-review loop the ratio is 12 to 7. | Review §3.2 | +| Assurance | No eval result blocks a merge, a release or a promotion. The ~4,900-case test suite is not a required check. | Review §3.5; live ruleset | + +The clearest sign is that fixes relocate failures instead of removing them. Channel tags fixed +unversioned consumers and produced merged fixes that do not run. The maintainer-comment gate fixed +unanswered comments merging and produced at least fourteen deadlock issues in four weeks. Claim +verification fixed phantom "Fixed" replies and produced guards that retract real fixes. Each repair +was sound locally. The pattern says the problems are structural, and structural problems need a +decided structure. That is what the accompanying ADR proposes; this PRD states what the structure +must achieve. + +The pattern is not peculiar to this org. Across about 22,000 developers, periods of high agent +adoption showed median review time five times longer, incidents per pull request up 243% and +deployments per week down 11.7%, while task throughput rose 34%. In a sample of merged agent pull +requests in significant public repos, 56% had no human reviewer ([market research](../research/market-agentic-software-factories-research-2026-10-10.md), §2.3). +Output is up across the industry and delivery is not. Verification is the bottleneck everywhere. + +Some things did improve and must be kept: hard failures are 0.9% of runs, against 100% failure +streaks in April; the self-mention loop that produced the PR 860 runaway has not recurred; trunk +corruption stopped once its guard became a required check. + +## 3. Users and journeys + +### 3.1 Users + +- **Maintainer.** The one human in the org: a technical solo builder running a portfolio of + products. States intent, makes the decisions the factory may not make, and is the operator of + last resort. Has no on-call rotation and no second reviewer. Can read code, owns the repos and + the deployments, and pays for model access from a personal budget. +- **Repo owner.** Adopts the factory in a repo and expects it to work without learning its + internals. Today this is the Maintainer in a different seat. +- **Factory developer.** Changes the factory itself. Usually an agent role working an issue; + sometimes the Maintainer in an interactive session. +- **Outside contributor.** Anyone on the internet, since every repo is public. Opens issues and pull + requests. Is not trusted. + +Roles (the agents) are actors in the system, not users of it. + +**Non-users.** Founders who cannot read code and ship from prompt-to-app builders are the largest +group of solo builders and cannot operate a Factory built on GitHub and CI; they are out of scope by +construction. So are teams looking for a hosted product with a vendor behind it ([market +research](../research/market-agentic-software-factories-research-2026-10-10.md), §3.1). + +**What the wider segment says it needs** matches this Maintainer's jobs below: a stopping rule that +can be trusted when nobody is watching, a hard cap on spend, review that scales past one person, and +guardrails that do not need a platform team. One person supervises two to five agents, with a +ceiling near eight, and the limit they name is their own attention (§3.1 to §3.3 of the research). + +### 3.2 Jobs to be done + +- When I have an idea, I want to approve it once and see it delivered, so that my time goes to + deciding what to build. +- When something is stuck, I want the item itself to tell me why and what would move it, so that I + do not have to investigate. +- When I say stop, I want everything to stop, so that I can trust the factory unattended. +- When I change the factory, I want to know when the change is live and to undo it quickly, so that + improving the factory is not riskier than leaving it alone. +- When I look at the bill, I want to see cost per delivered change, so that I can decide what is + worth automating. + +### 3.3 Key user journeys + +- **UJ-1. The Maintainer turns an approved idea into merged changes with two decisions.** + The Maintainer approves an idea and later arms the resulting epic. Stories are implemented, + reviewed and merged in dependency order. The Maintainer is asked nothing further unless a story + needs a decision. Climax: the epic closes with every story merged and each merge carrying an + independent approval. Edge: a story exceeds its cost ceiling; it is held with the amount spent, + the reason, and the choice required. +- **UJ-2. The Maintainer asks why a pull request has not merged.** + The Maintainer opens the pull request. One status block states the current state, the gate holding + it, the evidence and the next actor. Climax: the answer takes under a + minute and requires no script, log or API call. +- **UJ-3. The Maintainer stops the factory.** + The Maintainer sets one switch. No new agent run starts anywhere in the org; runs in flight finish + or checkpoint. Every affected item shows that it is paused and by whom. Releasing the switch + resumes work with no re-triggering. Edge: the factory pauses itself on budget; that pause is + labelled as automatic and clears itself, while a human pause never does. +- **UJ-4. The Maintainer steers work in flight.** + The Maintainer comments on an issue or requests changes on an agent's pull request, as they would + with a human colleague. The next run includes that input, and the input receives a recorded + disposition. Climax: the Maintainer can block an agent's pull request with an ordinary review. +- **UJ-5. A Factory developer ships a change to a role.** + The change merges. The pull request states which release carries it and when each ring will run + it. Promotion happens without the Maintainer when evals and ring evidence pass. Edge: a ring shows + a regression; promotion stops, the previous release keeps serving, and one action restores it + everywhere. +- **UJ-6. A Repo owner adopts the factory.** + The Repo owner adds one registry entry, and nothing to the repo itself. Within an hour every role serves the repo. No + later change to the factory requires an edit in that repo. +- **UJ-7. A Factory developer adds a role or gives a role a new trigger.** + One pull request in one repo adds a manifest, a prompt and eval cases. It reaches consumers by the + normal release path. No workflow file changes. +- **UJ-8. Capacity runs short.** + Budget for the week is 80% spent with three days left. The factory moves low-priority work to a + cheaper tier, defers the rest with a visible reason and re-evaluation time, keeps merge-unblocking + work running, and tells the Maintainer once. +- **UJ-9. The Maintainer reads the weekly scorecard.** + One page: changes delivered, lead time, share delivered without intervention, cost per change, + share of cost that was rework, items held and why, eval scores, release lag. Every number links to + the runs behind it. + +## 4. Glossary + +- **Factory** — the whole system: Roles, Controller, Harness, Kernel, Ledger, release tooling. +- **Work item** — an idea, epic, issue or pull request the Factory acts on. Exactly one Lifecycle + applies to each Work item. +- **Lifecycle** — the declared set of States and Transitions for a type of Work item. +- **State** — the single current position of a Work item in its Lifecycle. +- **State record** — the machine-readable account of a Work item's State, the reason for it, and the + facts the Factory owns about the item (attempts, spend, dispositions). It is computed at each + evaluation and published in the control room; it is a view, not a store (ADR-0013, decision 2). +- **Role** — a named agent responsibility (for example dev-lead, pr-review, qa-lead) defined by a + Role manifest. +- **Role manifest** — the data that defines a Role: the Intents it handles, its inputs, tools, + Engine tier, permitted Effects, budget class and evals. +- **Intent** — one kind of task a Role performs on a Work item (for example implement-issue, + fix-review-findings, review-pull-request). +- **Run** — one execution of one Intent by one Role on one Work item. +- **Task** — the typed input a Run receives. +- **Result** — the typed output a Run returns. +- **Effect** — a change to the outside world requested by a Result: a pushed commit, a comment, a + review, a label, a resolved thread. Starting a Run is not an Effect; only Admission does that. +- **Engine** — an LLM runtime and model that executes a Run. +- **Harness** — the component that prepares a Task, runs the Engine in a sandbox, validates the + Result and applies its Effects. +- **Controller** — the component that carries out Admission: it reads the facts, asks the Kernel + which Run to admit next, and starts it. +- **Admission** — the decision to start, defer or refuse a Run, made by the Kernel and carried out + by the Controller. +- **Kernel** — the Factory's deterministic decision logic, independent of GitHub and of any Engine. +- **Gate** — a condition a Work item must satisfy to make a Transition. +- **Hold** — a human or automatic stop on one Work item. +- **Pause** — a stop on all Runs in a scope (org, repo or Role). +- **Override** — a logged human action that bypasses a Gate. +- **Disposition** — the outcome for one Finding: fixed, declined, deferred or not applicable, + proposed by the author and accepted or rejected by the Role that owns the Finding's category. It + is read from the Finding's surface and is not kept as a separate record. +- **Ledger** — the durable, append-only record of Admissions, Runs and Effects: the Factory's run + records and artifacts in GitHub Actions, indexed in the control room. +- **Release** — one immutable version of everything that determines Factory behaviour. +- **Channel** — a named pointer to a Release that consumers follow. +- **Ring** — a set of Consumer repos that receive a Release at the same stage. +- **Consumer repo** — a repo the Factory serves. +- **Hosting repo** — the repo that contains the Factory. It is also a Consumer repo. The POC is + hosted in `petry-projects/.github-private`, and every measurement in this document that says + "hosting repo" was taken there. The MVP is built in a new repo, which becomes the Hosting repo; + `.github-private` then becomes a Consumer repo. +- **Advisory reviewer** — a third-party review bot whose output the Factory consumes but does not + control. An Advisory reviewer is a Lens. +- **Lens** — any source of judgement on a Work item: a Role, an Advisory reviewer, a tool such as a + static analyser or a test suite, or a human. +- **Finding** — one structured record of something a Lens observed: its source, a stable identity, + a location, a category, a severity and the content version it applies to. Whether it is open is + read from its surface. +- **Council** — the set of Lenses required for a class of Work item at a stage of its Lifecycle. +- **Convergence** — the condition in which every Lens in the Council has reported on the current + content version and no blocking Finding is open. + +## 5. Constraints + +These are facts of the environment. A design that ignores one is not viable. + +- **C-1. One Maintainer, no on-call.** The Factory must be safe unattended and must fail to a Hold, + not to an action. +- **C-2. GitHub Free plan, all repos public.** There are no org-level rulesets. Rulesets and the + merge queue work only because repos are public. Everything the Factory writes is world-readable. + Actions minutes are free. +- **C-3. LLM capacity is a subscription window, not a metered API.** The binding limit is a 5-hour + and a weekly window. The only usage signal is an unsupported endpoint that refuses the credential + CI holds. This basis is not secure: the provider announced in May 2026 that programmatic use + would move off subscription limits to API rates, and paused the change on the day it was due + (C-11). +- **C-4. GitHub-native operation.** The June 2026 release initiative fixed "no new repos, no + external infrastructure". The Maintainer lifted the first half on 2026-10-10: the Factory is + built in a new repo (§11, OQ-5). `[ASSUMPTION]` The second half, no always-on infrastructure, + still holds (OQ-1), with one exception: a small stateless relay that turns webhooks and a timer + into requests for a pass. +- **C-5. The platform loses events.** Scheduled ticks are dropped about 60% of the time and always + run the default branch. A concurrency group holds one running Run and, by default, one pending + Run; a third arrival evicts the pending one. GitHub now documents `concurrency.queue: max`, which + holds up to 100 pending Runs; no workflow here uses it. Events authored with the default workflow + token start no downstream workflow, except `workflow_dispatch` and `repository_dispatch`, and + those only within the same repo. +- **C-6. Identity rules.** A code owner must be a user or team, so a GitHub App cannot satisfy a + code-owner review. GitHub rejects an approval or a change request by a pull request's own author. + An approval by the last pusher is discarded under the last-push rule. `[ASSUMPTION]` GitHub's + terms permit one machine-user account per person, and the org already has one. +- **C-7. Self-hosting.** The Factory reviews and merges changes to itself. +- **C-8. Advisory reviewers are unreliable.** Most run on free or trial tiers and refuse work when + out of quota: 553 refusals in one week across six bots. +- **C-9. Twenty runner slots.** The free plan allows 20 concurrent hosted-runner jobs. + `[ASSUMPTION]` The limit is shared by every repo in the org. One push to a pull request in the + Hosting repo starts about 56 jobs. +- **C-11. Capacity terms can change on short notice.** In 2026 one platform moved its coding + product from flat to token-metered billing, one model provider announced and then paused the same + for automated use of its subscriptions, and another withdrew its command-line tool from + subscribers with about thirty days' notice. Limits on flat plans are unpublished and changeable. + The Factory's cost per change at API rates is several times its cost on a subscription + ([market research](../research/market-agentic-software-factories-research-2026-10-10.md), §2.5). +- **C-10. One Maintainer's attention is the first limit on output.** Each change that needs a human + audit, a cleared Hold or an Override costs Maintainer time, estimated at 10–12 hours a week at + the POC's 30–36 merges. `[ASSUMPTION]` The estimate is the Maintainer's rough figure. No requirement below can be met by spending more of it. + +## 6. Functional requirements + +### 6.1 Work-item lifecycle + +**Description.** Today a Work item's state is a tuple recomputed on every run from labels, about +eighty kinds of HTML-comment marker, review objects, check results and the head commit. No file +declares the states or the transitions. This feature makes the Lifecycle a declared, single thing. +Realizes UJ-1, UJ-2. + +- **FR-1. Declared Lifecycle.** The Factory defines one Lifecycle per Work-item type as versioned + data: its States, its Transitions, the Gate on each Transition and the actor allowed to make it. + - Every Transition the Factory performs in production is a declared one. An undeclared Transition + is detected automatically and reported as a defect. + - The Lifecycle definition is the source for documentation and tests; it is not restated by hand + elsewhere. +- **FR-2. One visible State.** Every Work item under Factory control shows, on the item, its current + State, the reason, the Gate it is waiting on with evidence, and the next actor. Every open Work + item is evaluated on every pass, so there is no next evaluation time to show. + - "Why is this pull request not approved?" is answerable from the item alone. + - A disagreement between the visible State and the State record is detected and repaired at the + next evaluation, and counted. +- **FR-3. Liveness.** No Work item remains in a State that is neither terminal nor held without a + scheduled next evaluation. + - An item with no evaluation for longer than its State's limit is escalated automatically. +- **FR-4. Terminal truth.** "Done" is derived from repository state: a merged change with a + non-empty diff that closes the item. + - A pull request with an empty net diff cannot close an issue. + - A completion statement appears only after the fact it reports is true. +- **FR-5. No lost work.** A request for work survives a dropped event, an evicted or cancelled Run, + a timeout and a failed Run. + - After any of these the item is re-evaluated without human action within the liveness limit. + - A cancelled Run is never read as "nothing to do". +- **FR-6. Resumable Runs.** A Run interrupted before completion preserves its progress, and the next + Run for that Intent resumes from it or records why it could not. Deferred past the first slices + by the Maintainer on 2026-10-11: until then an interrupted Run starts again. +- **FR-7. Idempotent by content.** Repeating a request for unchanged content causes no new agent + work. Content identity, not the commit hash, decides "unchanged". + - Updating a pull request's base without changing its own diff triggers no re-review and + invalidates no verdict. + +### 6.2 Roles and the run contract + +**Description.** The POC has five different ways of invoking a model, four ways of building a +prompt and seven ways of returning output. In the two busiest Roles the model process holds a write +credential, and roughly four thousand lines of harness exist to police what it then claims. This +feature gives every Role one contract. Realizes UJ-7. + +- **FR-8. Role as data.** A Role is defined entirely by its Role manifest, prompts and eval cases. + - Adding an advisory Role, or giving a Role a new Intent or trigger, changes no workflow file and + no Kernel code. + - The manifest is the only place a Role's triggers, identity, permissions and budget class are + written. +- **FR-9. One run contract.** Every Run receives a Task and returns a Result. Both are typed and + schema-versioned. + - A Result that fails validation fails the Run. No Effect from it is applied. + - A Task contains everything the Run needs; the model does not discover its inputs by exploring + the environment. +- **FR-10. The Harness applies every Effect.** The model process holds no credential that can write + to GitHub or push to a repository. + - This holds for every Role, including those that write code. + - No rule about what a Role may write is enforced by prompt text alone. +- **FR-11. Evidence before publication.** A statement by a Role about its own work is published only + after the Harness has verified it against repository state. + - "Fixed in commit X" is published only if X was produced by that Run and is on the remote branch. + - An unverifiable statement is not published and the Run is recorded as incomplete. +- **FR-12. Untrusted content is data.** Text written by an Outside contributor, an Advisory reviewer + or any other Role cannot change a Run's Task, permissions or Result schema. + - An injection test suite covers every Role and runs before a Release is promoted. + - A Run whose Task contains text from an Outside contributor or an Advisory reviewer has no shell + and no network tool. +- **FR-13. Engine independence.** A Role manifest names a capability tier, not a vendor or model. + - Changing the Engine behind a tier is a configuration change gated by that tier's evals. + - A Run records the Engine and model that actually served it, including after a fallback. +- **FR-14. Shared-branch safety.** A Role that pushes to a branch other actors push to never + discards a commit it did not make, never alters an existing test without a cited justification, + and never pushes a change that turns a passing test red. + +### 6.3 Orchestration + +**Description.** Runs are started today by thirty-odd workflows reacting to events, each with its own +conditions. One comment starts three workflows; one review starts four. Concurrency groups act as +the work queue and hold one item. Fifteen further workflows find and restart what fell through. +Realizes UJ-1, UJ-3, UJ-8. + +- **FR-15. One Admission point.** Every Run is admitted by the Controller. Admission evaluates Pause, + Hold, budget, duplication and concurrency in one place. + - There is no way to start a Run that bypasses Admission. + - Every Admission decision that differs from the previous one for that Work item, including a + refusal, is recorded in the Ledger with its reason. Repeats are counted, not recorded. +- **FR-16. One writer per Work item.** At most one Run that can produce Effects is active per Work + item. Requests that arrive during a Run are coalesced and evaluated when it ends. +- **FR-17. Bounded automation per Work item.** The number of automated actions on a Work item since + the last human decision has a ceiling. + - No agent action resets the count. The count does not depend on which account an agent uses. + - Reaching the ceiling places a Hold and notifies the Maintainer once. +- **FR-18. No self-trigger.** No Role's output can cause that Role, or a cycle of Roles, to run + again without passing Admission. The Lifecycle definition is checked for this before merge. +- **FR-19. Tolerance of event loss.** Correctness does not depend on the delivery of any single + event or timer tick. Losing one delays a Work item by at most the liveness limit. +- **FR-20. Bounded fan-out.** An event on one Work item causes agent work on other Work items only + where their content is affected. + - One merge to a protected branch causes no agent Runs on pull requests it does not conflict with. +- **FR-21. Priority under constraint.** When capacity is short, the Controller admits Runs in a + declared priority order: work that unblocks a merge first, then security, then the rest. + - The Maintainer's attention is capacity too. When more items await the Maintainer than a declared + number, only work that unblocks a merge is admitted. + +### 6.4 Human control + +**Description.** The Maintainer's controls differ by Role. One label has four meanings. There are +three mention grammars. The switch documented as pausing the fleet pauses one Role. Removing a hold +resumes nothing until another event happens. And because the Maintainer's account is also the +account two agent families act as, the Maintainer cannot review their pull requests and the +Factory's loop bound cannot tell them apart. Realizes UJ-2, UJ-3, UJ-4. + +- **FR-22. One control vocabulary.** Start, steer, hold, release, hands-off, re-evaluate and + override work the same way for every Role and every Work-item type, and are documented in one + place. +- **FR-23. A Hold binds everywhere.** A Hold on a Work item stops every Role's Effects on it from + the next Admission onward. + - Removing a Hold resumes work without any further action. + - Each Hold records who placed it and whether it needs a decision or will clear itself. +- **FR-24. Pause.** One switch stops Admission of all Runs in a scope: org, repo or Role. + - Paused items show that they are paused, by whom and since when. + - A Pause set by automation is distinguishable from one set by a human, and only the former is + cleared by automation. +- **FR-25. Steering reaches the Role.** Human input on a Work item, whether a body edit, a comment + or a review, is part of the next Run's Task. + - Every human comment on a Work item under Factory control is acknowledged by the Role that + read it. A plain comment steers and gates nothing; a review gates (FR-33). +- **FR-26. A human is never mistaken for an agent.** Every agent action is attributable to an agent + identity by actor alone. The Role that acted is recorded in the Ledger and shown in the output; + it need not be a separate actor. + - The Maintainer can approve, or request changes on, any agent-authored pull request. + - No Gate distinguishes human from agent by parsing comment text. +- **FR-27. Escalation states the decision.** When the Factory needs the Maintainer it says, on the + item, what decision is required and shows the evidence, and it notifies through a pushed channel. +- **FR-28. Override is a first-class action.** Forcing a review, merging without an approval and + promoting past a failed Gate are explicit actions that record who, when and why. + - None requires a hand-written API call. + - Every merge without a standing independent approval has an Override record. + +### 6.5 Review and merge integrity + +**Description.** The largest failure class in the issue history is the approval that never issues +or issues and is withdrawn: 63 issues, 45 of them in the last four weeks. Five independent gates can +each withhold or dismiss the single approval the ruleset needs. Realizes UJ-1, UJ-2. + +- **FR-29. Independent review.** No change merges on the say-so of its author's identity. The + reviewer identity differs from the author identity for every agent-authored change. +- **FR-30. Gate integrity.** A Gate cannot be cleared by the Role whose work it gates. +- **FR-31. Durable approval.** An approval is bound to the content it reviewed. + - It survives events that do not change that content. + - It is withdrawn only by new content or a new unresolved finding, and a withdrawal removes the + pull request from the merge queue. + - A Role never dismisses its own approval and then declines to re-review. +- **FR-32. Advisory reviewers are integrated through a contract, not a discussion.** Each Advisory + reviewer's registry entry names the structured surface its Findings are read from: a check run + and its annotations, code-scanning alerts, or review threads. + - A comment body is presentation for people. Nothing parses one to make a decision, waits for + one, or is blocked by one. + - An Advisory reviewer with no structured surface gates nothing. + - Only a Lens in the required Council gates or is waited for. An Advisory reviewer outside it is + never waited for: its absence, a quota notice or a trial expiry blocks nothing. + - A Finding from an Advisory reviewer outside the required Council gates only when the in-house + reviewer adopts it as its own. The in-house reviewer resolves each thread it does not adopt, so + that a platform rule requiring resolved threads stays in force. +- **FR-33. One Disposition per Finding per content version.** Every Finding from any Lens is + dispositioned exactly once for a given content version. The Gates read the outcome from the + Finding's surface: a thread, and which App resolved it. + - A Finding's identity is a structured key from its surface (a thread, an alert, an annotation), + never text matched in a body. An edited summary comment is therefore not a new Finding. + - A review thread, or a review that requests changes, from a person with write access is a + Finding from a human Lens and always gates. A thread from anyone else is handled as an Advisory + reviewer's would be. +- **FR-34. Serialized merges.** Merges to a protected branch are serialized and tested against the + actual merge result. + +### 6.6 Resource governance + +**Description.** Capacity is discovered by failing. The proactive headroom check reads zero on every +run because it probes with a key that is never supplied. The budget breaker was built, closed as +done, and cannot fire. The fleet alone uses more GitHub API requests per hour than the shared limit +allows. Realizes UJ-8, UJ-9. + +- **FR-35. Capacity is checked at Admission.** Before starting a Run the Controller checks remaining + LLM budget, GitHub API quota and concurrency. + - A Run that cannot be afforded is deferred with a visible reason. + - The capacity signal is exact where the platform provides one and a stated estimate where it + does not (C-3). A missing or failed reading defers the Run and raises an alert; it is never read + as "plenty". +- **FR-36. Graceful degradation.** Declared thresholds move the Factory through stages: full + service, cheaper tier for low-priority work, priority work only, Pause. Each change of stage + notifies the Maintainer once. Deferred past the first slices by the Maintainer on 2026-10-11: + until then the Factory has full service, deferral on an observed capacity fault, and Pause. +- **FR-37. Quota isolation.** The Factory's GitHub API usage and the Maintainer's interactive usage + do not draw on the same limit. The Factory's LLM usage is accounted separately from interactive + use; where both draw on one subscription window, the estimate in FR-35 says so. +- **FR-38. Complete cost attribution.** Every Run records tokens, cost, duration and outcome, + including Runs that failed, timed out or were cancelled, attributed to Work item, Role, Intent, + repo and Release, and marked as first-pass or rework. A Run killed before it can report is costed + by estimate from its duration, and marked as an estimate. +- **FR-39. Cost ceiling per Work item.** Each class of Work item has a declared ceiling. + Reaching it places a Hold that states the amount spent. The ceiling that gates is a number of + Runs, because decisions read GitHub only and what a Run cost is known only to the control room + (ADR-0013, decision 2). + +### 6.7 Observability + +**Description.** State is not observable today: diagnosing one stuck pull request meant correlating +timeline events by hand. The cost report omits the Hosting repo and nearly all pr-review calls. The +daily fleet failure report stopped three weeks ago and nothing noticed. 45% of all issues in the +Hosting repo are machine-filed reports. Realizes UJ-2, UJ-9. + +- **FR-40. Ledger.** Every Admission, Run and Effect is appended to a durable Ledger retained for at + least 90 days and queryable by Work item, Role, repo, Release and time. The Maintainer set 90 + days for the Factory's own records on 2026-10-10, in place of 180: it is how long GitHub keeps + run records in a public repo. The control room may keep what it has collected without limit, for + reporting. +- **FR-41. Work-item timeline.** The full history of a Work item can be reconstructed from the + Ledger alone. +- **FR-42. Scorecard.** A weekly and on-demand scorecard reports the success metrics in §9 from the + Ledger. +- **FR-43. Pushed alerts, watched watchers.** A breach of a liveness, capacity or integrity limit + notifies the Maintainer. Every monitor has a dead-man check: a monitor that stops reporting is + itself reported. +- **FR-44. Bounded report noise.** A recurring report updates one tracking item. It does not open a + new issue per run. + +### 6.8 Release and distribution + +**Description.** Two opposite regimes coexist: changes to dev-lead and pr-review are inert until a +tag moves, while persona prompts and manifests are live for every repo the moment they merge. The +version name is per agent but points at a whole-repo commit. Promotion reads run conclusions, which +the operating rules call unreliable. "Which agents exist" is written in eight places; a persona's +trigger surface in up to eight. Realizes UJ-5, UJ-6, UJ-7. + +- **FR-45. One Release unit.** A Release is one immutable version of every artifact that determines + behaviour: workflows, Kernel, Harness, prompts, Role manifests and configuration. + - Every Run records the Release it ran. + - No behaviour-determining artifact is fetched from a moving reference at run time. +- **FR-46. One path to production.** Every artifact type reaches Consumer repos through the same + staged path. The only artifacts live at merge are entry points that contain no logic, because the + platform always runs dispatched and scheduled workflows from the default branch. +- **FR-47. Promotion on evidence.** A Release advances a Ring only when eval scores for the + production Intents pass and the previous Ring shows verified outcomes: artifacts confirmed, not + run conclusions. + - A Ring with no Runs of the candidate does not count as evidence. +- **FR-48. Known time-to-live.** After a merge the Factory states which Release carries the change + and when each Ring will run it. A merged change not in production after the declared bound raises + an alert. +- **FR-49. Rollback.** One action restores the previous Release for all Consumer repos within 15 + minutes. +- **FR-50. Self-hosting safety.** The Release that reviews and merges a change to the Factory was + cut before that change existed, so no change reviews itself. The Factory's own repo is served by + `next`. A Release reaches `stable`, and so any Consumer repo, only on evidence from `next`. +- **FR-51. Onboarding.** A Consumer repo adopts the Factory with one registry entry and no file of the Factory's, and + is served by every Role within an hour. A later Release, new Role or new trigger requires no edit + in the Consumer repo. Installing Advisory reviewers in vendor consoles is outside this bound. +- **FR-52. Version-skew safety.** Messages between components running different Releases carry a + schema version. A sender does not emit a message its receiver cannot read. +- **FR-53. One fact, one place.** The set of Roles, the set of Consumer repos, each repo's Ring and + each Role's triggers are each written once. Everything else is derived. + +### 6.9 Assurance of the Factory itself + +**Description.** The test suite is large (about 4,900 cases, 1.46 lines of test per line of script) +and is not a required check. Evals for dev-lead and pr-review score their mention-advisory prompts, +not the flows that write code and approve pull requests. Many tests run against a `gh` stub that +accepts any flags. The suite cannot run on the Maintainer's machine. Realizes UJ-5. + +- **FR-54. Verification blocks merge.** The Factory's test, contract and validation suites are + required checks on the Hosting repo. +- **FR-55. Evals cover production Intents.** Every Intent that writes code or decides a Gate has a + held-out, scored eval set that exercises the production prompt and Harness path. + - "Held-out" names its mechanism. In a public repo it is either stored where the Role under test + cannot read it, or declared a convention guarded by code ownership. +- **FR-56. Real-interface tests.** Every interaction with GitHub and with an Engine is tested + against recorded real responses or a sandbox, so that a call the real interface rejects fails a + test. +- **FR-57. Lifecycle verification.** The declared Lifecycle is checked mechanically for + termination, absence of undeclared cycles, and Hold and Pause semantics. +- **FR-58. Local reproducibility.** A Factory developer can run the full suite, and replay a + recorded Work item through the Kernel, on macOS and on Linux with documented prerequisites. +- **FR-59. Acceptance the author cannot edit.** Every code-writing Intent is checked against at + least one acceptance signal that the authoring Role can neither read in advance nor change. The + Consumer repo's own tests do not qualify, because the author can edit them. + - A Work item that leads to a code change carries an acceptance contract before the work is + admitted: criteria, each naming how it will be checked. It is drafted by a Role other than the + implementing one, as the first commit of the work's pull request, and fixed by the Maintainer's + approval at that commit. + - Passing evidence for every criterion, on the content to be merged, is a condition of + Convergence. + - At first a Role's judgement is allowed as a check. Executable and held-out checks are required + of the highest-priority criteria as measurements allow. + +### 6.10 Security + +**Description.** One owner credential serves code-writing, cross-repo dispatch, labelling, persona +comments and fallback authentication. Reusable workflows inherit every org secret. One gate fetches +and executes a script from another repo's default branch at run time. Realizes UJ-3, UJ-6. + +- **FR-60. Least privilege per Role.** Each Role's Effects are applied with a credential scoped to + that Role's permitted Effects and short-lived. No Role uses the Maintainer's credential. + - Where the platform requires a user account (C-6), that one long-lived credential is confined to + a job that runs no model and no code from the Work item. +- **FR-61. No unpinned execution.** The Factory executes no code and reads no policy from a mutable + reference outside its own Release. +- **FR-62. Public by default.** Every Effect is written as world-readable. Secret values never + appear in logs, comments or the Ledger. +- **FR-63. Outside contributions.** A pull request from an Outside contributor never receives a Run + that holds secrets or can produce write Effects without a Maintainer decision. That decision + lapses when the contributor pushes again. + +### 6.11 Convergence: the council of Lenses + +**Description.** The POC already runs a council: a review cascade with four specialist prompts and +a second-opinion model from another vendor, nine advisory personas, and nine Advisory reviewers. +They interact by posting comments at each other and at the author. The result was 343 review +dismissals on 150 merged pull requests, commits per merged pull request rising from 2–4 to 7–10, +and a scorecard that cannot say which Lens caught anything. This feature keeps the council and +replaces the discussion with a contract. Realizes UJ-1, UJ-2, UJ-9. + +- **FR-64. One Finding contract for every Lens.** Roles, Advisory reviewers, tools and humans all + contribute Findings in one schema. What a person reads on the Work item is rendered from those + records by the Harness. + - A Role that reviews declares a rubric: its criteria as data. Each of its Findings cites a + criterion, and its Result reports on every criterion that applies. + - A rubric is a minimum. A Finding outside it is allowed and counted, and rubrics grow from what + they missed. +- **FR-65. Lenses report; they do not converse.** No Lens's output triggers another Lens or is + addressed to one. Each reports to the Controller. + - A Lens that judges a change gives its Findings without seeing another Lens's verdict on that + content version. +- **FR-66. One author per change.** Lenses judge a change. They do not co-write it. A Finding is + acted on by the Role that authored the change. +- **FR-67. Convergence is decided.** The Kernel decides Convergence from Findings and their + Dispositions. It is not inferred from a conversation going quiet. + - A Work item that has not converged within its declared bound of rounds or Runs is placed on + Hold with the open Findings listed. +- **FR-68. A Council is declared per class of Work item and stage.** Which Lenses are required for + an idea, a plan, a design, a code change and a Release is data, and is not the same set for each. + - Work that divides into independent parts (research, analysis, review) may use Lenses in + parallel. Work that is one interdependent piece (implementation) has one author. +- **FR-69. A Lens earns its place.** The Ledger records, per Lens, the Findings it raised, the share + that were acted on, the share no other Lens raised, and its cost. + - A Lens joins a required Council only with a measured marginal contribution, and leaves it when + that contribution falls below a declared level. + - Independence is measured, not assumed: two Lenses that raise the same Findings count as one. +- **FR-70. Lenses differ in what they can see, not only in which model they are.** A Council for a + code change includes at least one Lens grounded in execution (tests, static analysis, a running + system) and at least one acceptance signal the author cannot edit (FR-59). + +- **FR-71. Feedback moves left; the verdict does not.** Every Lens that can run before a push runs + inside the Run, and its Findings go back to the author for a bounded number of repair turns + before anything is pushed. + - The same Lens runs again where the author controls nothing, and only that run counts toward a + Gate. + - Both runs use the same pinned tools and configuration, defined once. + - A Run whose work still fails after its repair turns keeps the work and reports what failed. It + is not discarded. + - An acceptance signal the author must not see (FR-59) is never fed back. + +### 6.12 Sourcing and capacity basis + +**Description.** Two requirements from the market research. The workers the POC built by hand are +now offered by ten or more vendors, and the subscription the POC runs on is the usage its provider +tried to move to metered rates in 2026. Realizes UJ-7, UJ-8. + +- **FR-72. Workers and reviewers are adopted, not rebuilt.** Where a maintained product or toolkit + performs an Intent or acts as a Lens, the Factory uses it through the run contract or the Finding + contract. The Factory builds a worker of its own only where none meets the contract. + - Replacing one worker or reviewer with another changes configuration, not the Kernel. + - No part of the Factory depends on a product that has no declared replacement. +- **FR-73. The Factory can run on metered capacity under a hard cap.** Every Role can be served from + a metered API credential as well as a subscription, and the choice is configuration. + - A hard monthly cap in currency stops Admission when it is reached. It is a brake outside the + Kernel. It applies when a metered credential is in use and is read from the provider's own + usage figure. On the subscription the brake is a ceiling on Runs. + - The scorecard reports what each period would have cost at API rates, whichever basis paid for + it. + +## 7. Non-functional requirements + +Targets marked `[ASSUMPTION]` are proposals for the Maintainer to calibrate. Baselines are from §2 +and the review document. + +| ID | Quality | Target | POC baseline | +|---|---|---|---| +| NFR-1 | Flow efficiency | At most 10 Runs per merged pull request; at most 2% of admitted Runs lost or started later than the liveness limit. Passes that admit nothing are reported separately and are not capped `[ASSUMPTION]` | About 1,500 workflow runs per merge including CI; 24% cancelled | +| NFR-2 | Lead time | Issue to merge p50 ≤ 8 h and p90 ≤ 48 h for items needing no human decision `[ASSUMPTION]` | p50 24.5 h, p90 296.5 h | +| NFR-3 | Autonomy | At least 85% of agent-authored pull requests merge with a standing independent approval and no human action outside declared decision points `[ASSUMPTION]` | At most 40% | +| NFR-4 | Gate integrity | 100% of merges carry either a standing independent approval or an Override record | 66% approved; no Override record exists | +| NFR-5 | Liveness | No Work item is without a next evaluation for more than 24 h | 22 open pull requests older than 7 days; three flagged as runaway daily | +| NFR-6 | Cost visibility | 100% of Runs attributed; rework share reported | Hosting repo and pr-review absent from the cost report | +| NFR-7 | Cost efficiency | Rework at most 20% of spend; trivial sync pull requests at most 5% of spend `[ASSUMPTION]` | Rework unmeasured; sync pull requests were 30% of one week | +| NFR-8 | Release lag | Merge to stable ≤ 48 h at p90 with no manual promotion, with candidates batched because every eval case spends the capacity in C-3; rollback ≤ 15 min | Manual override promotions on four dates in ten days | +| NFR-9 | Operator load | At most 30 minutes a week of Maintainer time on actions that are not decisions `[ASSUMPTION]` | Not measurable today; routine per the operating notes | +| NFR-10 | Defect inflow | Factory-defect issues at most 5 a week and falling `[ASSUMPTION]` | 21–36 a week | +| NFR-11 | Changeability | New advisory Role: one pull request and one App registration. New trigger for an existing Role: one pull request, live within NFR-8 | New trigger took 32 days and at least six pull requests across two repos | +| NFR-12 | Maintainability | Kernel decision logic is free of I/O; a size and complexity budget per module is enforced in CI | 17 functions over 150 lines; one script with 1,025 top-level lines | +| NFR-13 | Portability | The suite runs on macOS and Linux | Fails on stock macOS | +| NFR-14 | Quota headroom | Factory GitHub API usage stays under 60% of its own limit at p95 `[ASSUMPTION]` | 115–140% of a shared limit | + +NFR-3 and NFR-10 are a pair. On the POC's first-pass correctness they cannot both be met: at 85% +autonomy the share of merges that carry a defect must roughly halve to keep defects at five a week +(review document, §14.4). Autonomy is raised only as SM-C1 allows. + +NFR-1 counts Runs. The runner pool of C-9 also needs a budget: passes and nudges take at most 10% of +the org's job-minutes `[ASSUMPTION]`. + +## 8. Non-goals + +- The Factory is not a product for other organisations. It serves `petry-projects`. +- The Factory does not replace GitHub as the system of record for code, reviews and merges. +- The Factory does not remove the Maintainer from the decisions in §6.4: approving ideas, arming + epics, resolving escalations, overriding Gates. +- Defining what a change is judged against is not review depth, and is in scope (FR-59). +- The MVP does not improve review depth. Better bug-finding (symbol context, specialist routing, + few-shot from merged pull requests) waits until the line is stable, because its benefit cannot be + measured on a line that loses a third of its work. This non-goal sets the MVP's ceiling: once + approvals issue reliably, how often they are right is what limits autonomy (FR-59, and the note + under §7). +- The MVP adds no new Roles and no new Advisory reviewers. It gives the existing council a contract + and a measure (§6.11). Enlarging the council comes after the measure exists. +- The MVP does not provide 24-hour operations or a paging rota. +- The MVP does not target private repos or a paid GitHub plan. + +## 9. Success metrics + +### 9.1 Primary + +- **SM-1. Autonomy rate.** Share of agent-authored pull requests merged with a standing independent + approval and no human action outside declared decision points. Target per NFR-3. Validates FR-5, + FR-15, FR-29 to FR-33. +- **SM-2. Lead time.** Issue-to-merge p50 and p90. Target per NFR-2. Validates FR-3, FR-5, FR-7, + FR-19. +- **SM-3. Cost per merged change**, with rework share. Targets per NFR-6 and NFR-7. Validates FR-7, + FR-20, FR-35 to FR-39. + +### 9.2 Secondary + +- **SM-4. Runs per merged pull request and lost-Run share.** NFR-1. Validates FR-15, FR-16, FR-20. +- **SM-5. Unapproved merges without an Override record.** Target zero. Validates FR-28, FR-29. +- **SM-6. Items past the liveness limit.** Target zero. Validates FR-3. +- **SM-7. Release lag and manual promotions per week.** NFR-8. Validates FR-46 to FR-49. +- **SM-8. Factory-defect issues per week.** NFR-10. +- **SM-9. Maintainer non-decision time per week.** NFR-9. Validates FR-22 to FR-28. +- **SM-10. Eval coverage.** Share of code-writing and Gate-deciding Intents with a passing held-out + eval set. Target 100%. Validates FR-55. +- **SM-11. Rounds to Convergence.** Commits and review rounds per merged pull request. POC baseline: + 7–10 commits at the median. Validates FR-65 to FR-67. +- **SM-12. Lens contribution.** Per Lens: Findings acted on, Findings unique to it, and cost. + Validates FR-69. +- **SM-13. Factory against one agent.** Cost, lead time and escaped defects per merged change for + work run through the Factory, against the same class of work done by one agent with the same + pre-flight checks and one human approval. The market's strongest objection is that orchestration + does not beat a single well-run agent; this is the measure that answers it. Validates the + premise, not a requirement. + +### 9.3 Counter-metrics (do not optimise) + +- **SM-C1. Escaped defects.** Reverts and post-merge fixes per hundred merges. Counterbalances SM-1 + and SM-2: autonomy and speed must not be bought by weakening review. +- **SM-C2. Holds cleared by automation.** Counterbalances SM-1 and SM-6: liveness must not be bought + by auto-clearing human Holds. +- **SM-C3. Review skipped or carried forward incorrectly.** Share of merges whose final content was + never reviewed. Counterbalances SM-3 and SM-4: cost must not be bought by not reviewing. +- **SM-C4. Override count.** Counterbalances SM-5: an Override record must not become the routine + path. + +## 10. MVP scope + +### 10.1 In scope + +- The Roles that form the delivery line: dev-lead (implement issue, fix CI, fix review findings, + resolve conflicts), pr-review, and the initiative driver. +- The advisory personas as they exist, moved onto the common run contract and release path. +- The Hosting repo and at least two Consumer repos, served from one Release. +- Every requirement in §6 for those Roles and repos. +- The identity requirement (FR-26, FR-60) for every agent, including the planning Roles that + §10.2 otherwise leaves as they are. Exit criterion 5 depends on it. +- Migration of in-flight Work items without loss. +- `[ASSUMPTION]` Two Consumer repos are enough to prove multi-repo operation for the MVP. + +### 10.2 Out of scope for the MVP + +- Multi-repo initiatives. Deferred: they multiply every coordination problem above. +- MCP-powered review, review-depth work, self-improving skills, automated model onboarding. + Deferred: each needs the measurement the MVP provides. +- Feature ideation, idea triage and the initiative planner. They keep running as they are and move + to the common contract after the delivery line. +- Fleet-wide compliance automation and the spam guard. Owned by the standards repo. +- Report and health workflows that become redundant once the Ledger exists are retired, not ported. +- `[ASSUMPTION]` The planning Roles can stay on their current implementation during the MVP without + blocking it. + +### 10.3 Exit criteria: POC to production + +The MVP is in production when, for four consecutive weeks on the Hosting repo and two Consumer +repos: + +1. SM-1, SM-2, SM-5 and SM-6 meet their targets. +2. SM-3 is reported for 100% of Runs. +3. No Release was promoted manually and at least one rollback was exercised. +4. Every compensating workflow listed in the review document, §3.2, is retired or has a recorded + reason to remain. +5. The Maintainer's account authored no agent commit, comment or review. + +### 10.4 Build versus adopt + +From the competitive analysis ([market research](../research/market-agentic-software-factories-research-2026-10-10.md), §4.3). + +| Capability | Position | Why | +|---|---|---| +| The Lifecycle, Admission, the State record and Ledger | **Build** | No product offers a cross-repo account of why each Work item is waiting | +| The merge decision and its policy | **Build** | Every vendor hands it back; four refuse in writing | +| The Finding contract and Convergence | **Build** | No product draws reviewers from different vendors and kinds | +| Staged release of the Factory across a portfolio | **Build** | Not documented by anyone | +| Capacity admission and the hard cap | **Build** | Vendors meter spend; none admits work against what is left | +| Turning an issue into a pull request | **Adopt** | Offered by more than ten products | +| CI repair and review follow-up on an agent's own pull request | **Adopt** | Offered by more than eight | +| First-pass review | **Adopt** | Free on public repos from several vendors | +| Deterministic analysis | **Adopt** | Open tools reporting through code scanning | +| Requested effects with a separate applier | **Adopt if it fits** | GitHub's Agentic Workflows ships this pattern under an open licence (OQ-14) | + +## 11. Open questions + +1. **OQ-1. Is "no always-on infrastructure" still a hard constraint?** It decides whether an + off-GitHub control plane is a candidate for the MVP or a later option. One exception was allowed + by the Maintainer: a small stateless relay that turns webhooks and a timer into requests for a + pass, keeps no state and decides nothing. +2. **OQ-2. Will the Maintainer move the code-writing Role off the owner account?** FR-26 and FR-17 + need it. It reverses the choice made in `petry-projects/.github-private#1316 — dev-lead regressed + to committing as donpetry-bot (review-only machine user) instead of don-petry` and recorded in + the interaction model, §12.2. Decided by the Maintainer on 2026-10-10: yes. No agent runs as the + owner account. The Maintainer may still author changes to the Factory's repo under that account, + as a person. Each Role has its own GitHub App, so that its permissions are a ceiling the platform + enforces. A Role is given its App when it first needs to write; until then it reads through one + shared reader App. The review document, §6.4, had recommended one App. +3. **OQ-3. What supplies the LLM capacity signal, and does the Factory get its own LLM credential?** + FR-35 has no feed today. An estimate from the Factory's own usage is sound only if interactive + sessions do not spend from the same window unobserved. The analyst's comment on the idea + (`petry-projects/.github-private#2226 — Factory MVP: rebuild the agentic factory around a + declared control plane, one run contract and a council of Lenses`) sets out four credential + options and suggests moving the eval lane to metered access first. Decided by the Maintainer on + 2026-10-10: the Factory runs on a subscription credential of its own, not the Maintainer's, stored + as an environment secret in the Factory's repo and restricted to its default branch. The capacity signal therefore + stays an estimate, shared with interactive use and with the POC while both run. One observed + signal is added: a Run that ends because the window ran out says so, and Admission defers every + model Run until the window is expected to reset. +4. **OQ-4. Should code-owner review stay required with one human in the org?** It is the reason the + reviewer must be a machine user and not an App. +5. **OQ-5. Should the Factory have its own repo?** Decided by the Maintainer on 2026-10-10: yes. It + is built greenfield in a new repo and cut over through the channel tags `.github-private` + already has, one Intent and one Ring at a time (review document, §18). The repo is + `petry-projects/software-factory`. Open: what `.github-private` keeps once every caller is + re-pinned. +6. **OQ-6. Which Advisory reviewers are worth waiting for?** Four of seven gated reviewers never + participate. The rule was decided on 2026-10-10 (OQ-13): only a Lens in the required Council is waited + for, and a reviewer outside it gates only when the in-house reviewer adopts its Finding. +7. **OQ-7. Are the `[ASSUMPTION]` targets in §7 the right ones?** +8. **OQ-8. Where does the Ledger live and who may read it?** Everything in a public repo is public. + Decided by the Maintainer on 2026-10-10: the Ledger is the Factory's own run records and + artifacts, and `petry-projects/software-factory-control-room` is a public, rebuildable view of + them with a dashboard. Every panel is public. Held-out eval sets live under a different owner, + where no Factory App is installed, because a public repo cannot hide them from the Role they test. +9. **OQ-9. What is the budget for the migration itself**, in weeks and in LLM spend, given that the + POC must keep delivering while it is replaced? The three conditions ADR-0013 required were set + by the Maintainer on 2026-10-10 (ADR-0013, decision 10). POC defects are fixed as needed, with no + weekly cap. There is no dated limit and no automatic stopping rule: if the Factory is not adopted + and used, the Maintainer decides to stop, informed by the control room. Still open: the LLM + spend. +10. **OQ-10. Should third-party reviewer comments keep gating approvals?** Decided by the Maintainer + on 2026-10-10: reviewers are integrated through a contract, and structured Findings gate where + comment bodies do not (FR-32, FR-33). Later the same day: only required Lenses gate, and a + reviewer outside the required Council gates by adoption. The nine current reviewers' surfaces are established in the + review document, §16.2: one has a full contract, one a findings API, one a structured verdict, + and the rest review threads only or nothing. +11. **OQ-11. Is it worth trying the three cheapest changes first and re-measuring?** Stopping the + rebase fan-out, moving agents to their own identity, and narrowing what gates an approval cost + weeks. Four weeks of data after them would show how much of the rest is needed. +12. **OQ-12. What is the Council for each stage?** §6.11 requires it to be declared and measured. It + does not say which Lenses belong in it. The research gives a starting shape (review document, + §15.4), and §16.6 proposes a starting council for code changes: six deterministic and + execution-grounded Lenses on one contract, the in-house reviewer, and three or four of the + nine outside reviewers kept for measurement. +13. **OQ-13. Which outside reviewers are removed now?** §16.6 recommended removing four from the + required set (CodeAnt, Graphite, Copilot, Qodo) and funding or dropping a fifth (Codex). Each + is a subscription or an installation the Maintainer controls. Decided by the Maintainer on + 2026-10-10: compatibility with the structured contract decides, not volume. Copilot and Qodo + come off now, because neither is reviewing and neither shows a structured surface here. Codex + stays installed, unfunded and not required, so its refusals gate nothing. CodeAnt and Graphite + stay as review-thread Lenses and are measured with the rest. Graphite is a paid subscription. + Copilot may be re-enabled, and would then enter as a review-thread Lens under the same + measure. +14. **OQ-14. Is GitHub's Agentic Workflows adopted as the worker half of the run contract?** It is + the nearest substitute for what ADR-0013 proposes to build there. Its documentation does not + list a subscription credential for the Claude engine, which conflicts with C-3 until C-11 + forces the question anyway. +15. **OQ-15. What is the metered fallback budget?** FR-73 needs a monthly figure in currency at + which the Factory stops, and a decision on whether to hold a funded API credential in reserve. + Decided in part on 2026-10-11: a metered API credential is the fallback if the Factory's own + subscription cannot be used. The monthly figure is still open. +16. **OQ-16. Is release and operation the next scope after the MVP?** It is the widest gap in the + market at a solo builder's price and is outside §10.1. +17. **OQ-17. Should the first non-goal be restated?** The research supports keeping the Factory out + of the business of selling orchestration: attention there is large and willingness to pay is + low. "Not a commercial product; open for others to read and reuse" would say that more exactly + than "not a product for other organisations". + +## 12. Assumptions index + +This PRD was revised once after the architecture analysis tested it against the constraints. The +twelve changes and their reasons are recorded in the review document, §10. + +- §5 C-4 — "no always-on infrastructure" still holds. +- §5 C-6 — GitHub's terms allow one machine-user account per person. +- §7 NFR-1, NFR-2, NFR-3, NFR-7, NFR-9, NFR-10, NFR-14 — numeric targets are proposals. +- §5 C-9 — the 20-job limit is org-wide. §5 C-10 — the Maintainer-time estimate. +- §7 — passes and nudges within 10% of job-minutes. +- §10.1 — two Consumer repos are enough to prove multi-repo operation for the MVP. +- §10.2 — the planning Roles can stay on their current implementation during the MVP without + blocking it. + +## Appendix A. Failure classes to requirements + +Classes and counts are from the review document, §2.3. + +| Failure class | Issues (hosting repo / standards repo) | Requirements | +|---|---|---| +| A. Approval never issues, or issues and is withdrawn | 63 / 6 | FR-2, FR-29 to FR-33 | +| D. Work dropped between event and agent | 47 / 17 | FR-3, FR-5, FR-15, FR-16, FR-19 | +| B. Status reported but not real | 37 / 14 | FR-4, FR-10, FR-11, FR-47 | +| E. Quota and budget exhaustion | 33 / 6 | FR-35 to FR-39 | +| C. Merged fix is not what runs | 25 / 72 | FR-45 to FR-53 | +| F. Identity, credential, permission | 21 / 3 | FR-26, FR-29, FR-60 | +| G. Re-review churn, rebase treadmill, runaway | 17 / 4 | FR-7, FR-17, FR-20, FR-34 | +| I. Human steering does not reach the agent | 14 / 12 | FR-22 to FR-25 | +| J. Tests and evals not measuring the real thing | 14 / 16 | FR-54 to FR-58 | +| K. Automation that corrupts or destroys | 13 / 8 | FR-10, FR-14, FR-61 | +| H. Timeout loses the work | 9 / 3 | FR-6 | diff --git a/docs/research/market-agentic-software-factories-research-2026-10-10.md b/docs/research/market-agentic-software-factories-research-2026-10-10.md new file mode 100644 index 00000000..8dd7d472 --- /dev/null +++ b/docs/research/market-agentic-software-factories-research-2026-10-10.md @@ -0,0 +1,501 @@ +> First written in `petry-projects/.github-private`, on the draft pull request "docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed)" (number 2225, commit `7d7d04f`). It now lives in this repo. Citations of files and lines in `.github-private` are evidence about the current system and are left as written. + +# Market Research: agentic software factories + +**Status:** Complete — for maintainer review +**Date:** 2026-10-10 +**Research type:** Market +**Method:** BMAD market-research workflow (`bmad-market-research`), run by the analyst persona +**Priority segment:** solo founders and indie builders +**Inputs:** [architecture review](../architecture/factory-mvp-architecture-review.md) §13, §15, §16; +[PRD](../prd/agentic-software-factory.md) + +## Executive summary + +1. **"Software factory" became the category label in 2026, and almost every product that uses it + keeps a human at the merge.** Four vendors refuse in writing to merge to a default branch. The + no-human-review "dark factory" is one company's published practice. +2. **Writing code is now a commodity; verifying it is the bottleneck.** Issue-to-pull-request, + CI repair and first-pass review are each offered by eight or more products, several free on + public repos. The largest telemetry study found review time up fivefold and incidents per pull + request up threefold as agent output grew. +3. **The price of automation is moving from flat to metered.** Interactive use stays at $20 to + $200 a month. Unattended and CI use is being moved to API rates: GitHub did it on 1 June 2026; + Anthropic announced the same for its programmatic use and paused it on the day. A solo factory's + economics depend on which side of that line its workload falls. +4. **Model providers are absorbing the layer above them.** First-party agents overtook the + independent tool vendors within six months, providers are buying the toolchain, and more than a + dozen products in this space were shut, sold or renamed in a year. +5. **The solo builder's pains are the ones this org measured in itself:** review burden, usage caps + and unpredictable cost, work that passes tests and is wrong, and the attention it takes to + supervise. One person runs two to five agents, with a ceiling near eight. +6. **Attention at the orchestration layer is large and does not convert to revenue.** A dozen open + projects hold between 6,000 and 100,000 stars. The best-known commercial one, with thousands of + daily users, closed for want of a business model, and the best-known open one was disowned by + its author. +7. **Five things no product offers a solo builder today:** a governed, unattended merge on the + operator's own policy; release and operation at solo price; a cross-repo view of why each work + item is waiting; one versioned definition rolled out in stages across a portfolio; and admission + of work against a flat subscription with reviewers independent of the writer's vendor. +8. **The main barrier is unproven value.** Nothing shows that a factory beats one well-run agent + for a single person with a modest backlog. The burden of proof is on the factory. +9. **The nearest substitute is GitHub's own.** Agentic Workflows already ships the pattern in which + the agent requests effects and a separate job applies them, with spend ceilings, under an open + licence. On GitHub's documentation an agent-approved merge is one setting and one preview away. + +**What this means for the PRD** is in §7. In short: build the control plane and the merge decision, +adopt the workers and the reviewers, plan for metered capacity, and prove the factory against a +single-agent baseline before claiming it is better. + +## 1. Introduction and method + +### 1.1 Why this research now + +The architecture review treated this org's factory as a proof of concept and proposed a production +MVP. It compared seven systems on their design. It did not ask what the market offers a buyer, who +that buyer is, or what vendors will ship before the MVP is finished. This research does. + +### 1.2 Scope + +- **Topic:** products, platforms and open projects that promise to take software from intent to a + shipped, maintained change with agents doing most of the work. +- **Customer:** solo founders and indie builders: one person, one or many products, consumer or pro + subscriptions, no operations team and no second reviewer. +- **Geography:** global; English-language sources. + +### 1.3 Method and limits + +- Three parallel research passes on 2026-10-10: market definition, size and pricing; the customer; + and a competitive teardown from a solo builder's seat. +- Claims are marked **(P)** where the company's or author's own page was read, **(S)** where the + source is press or a third party, and listed in §8 where they could not be established. +- Vendor numbers are vendor claims. Every merge-rate and productivity figure found for a coding + agent is self-reported; no independent quality evidence for any agent's merged output was found. +- Pages were read through a fetcher that returns a model-extracted rendering. Figures should be + re-opened before being quoted elsewhere. +- Reddit could not be read. First-hand customer evidence leans on Hacker News, which skews technical + and sceptical. +- No survey isolates solo builders. Segment findings combine general developer surveys with + first-hand accounts. + +## 2. Market analysis and dynamics + +### 2.1 Definition and categories + +There is no settled name. Gartner renamed its coverage "Enterprise AI Coding Agents" in May 2026 +(S). Forrester uses "Agentic Development Platforms" (S). Practitioners say "software factory", +"dark factory" and "harness engineering". The categories below describe jobs, not companies; every +vendor in the first two now ships pieces of the others. + +| Category | What it does | Buyer | Examples | +|---|---|---|---| +| Assistants and AI-native editors | Completion, chat, an in-editor agent; a human per change | Developer; seat buyer | GitHub Copilot, Cursor, Devin Desktop, Kiro | +| Autonomous coding agents | Take a task, plan, edit, test, open a pull request, locally or in a cloud sandbox | Developer; engineering lead | Claude Code, Codex, Devin, Jules, Factory Droids | +| Orchestration and "software factory" platforms | Run many agents through intake, specification, build, verification and shipping | Head of engineering; small teams | Factory, 8090, Warp Factories, Harness Cosmos, Gas Town, Symphony | +| Prompt-to-app builders | A prompt to one hosted, deployed app | Non-developers, founders | Lovable, Replit, Bolt, v0 | +| Review and verification | Automated review, bug finding, fix loops | Engineering manager; maintainer | CodeRabbit, Greptile, Cursor Bugbot, Gitar, cubic | +| Agentic CI and control planes | Agents inside the forge with identity, permissions, budgets, audit | Platform and security | GitHub Agent HQ and Agentic Workflows, Claude Managed Agents, Codex cloud | +| Spec-driven development | The written spec is the durable artefact | Tech leads | Spec Kit, Kiro, Tessl, OpenSpec, BMAD | +| Internal factories | In-house systems cited as proof | Not for sale | StrongDM, Stripe, Ramp, OpenAI's internal harness | + +**Who says "software factory".** It is a product name at 8090 and Warp, and positioning at Factory, +Augment (now Harness), Tessl and others. All of those keep human gates at specification and at +merge. Only StrongDM publishes a practice with no human review (P). + +### 2.2 Size and growth + +Published estimates disagree by a factor of three or more, and none was opened directly (all S). + +| Horizon | Range | What explains the spread | +|---|---|---| +| 2025 | $4.0B to $11.8B | Enterprise application spend only, against anything "AI for developers" | +| 2026 | $9.4B to $16.1B | Scope and base year | +| 2030–2031 | $25.7B to $97.9B | Compound growth of 25% to 48% assumed | + +The forecasts look stale against stated revenue. Run-rates reported for a handful of leaders already +sum past $8B a year before GitHub Copilot and Codex, neither of which discloses revenue. + +| Company | Reported traction | Kind | +|---|---|---| +| Cursor | About $4B annualised, June 2026; reported acquired by SpaceX in an all-stock deal | Press, single source for the revenue figure | +| Claude Code | Above $2.5B run-rate, February 2026 | Press | +| Cognition (Devin) | Above $1B run-rate, September 2026; includes the former Windsurf line | Company | +| GitHub Copilot | 50M users including free tiers, July 2026; 4.7M paid, January 2026 | Company | +| Lovable | About $500M annual run-rate, June 2026 | Company | +| Factory | Raised at a $5B valuation, September 2026; revenue not disclosed | Company and press | +| 8090 | Raised $135M, June 2026; revenue not disclosed | Company | + +**No estimate sizes the orchestration or "software factory" category by itself.** + +### 2.3 Adoption + +- **Agents are the default at work.** 90% of professional developers use coding agents at least + weekly and 68% daily (JetBrains, more than 15,000 developers, mid-2026, P). +- **First-party agents overtook the tool vendors in six months.** Share using each at work, January + to mid-2026: Claude Code 18% to 39%, Codex 3% to 16%, Cursor 18% to 12%, Copilot 29% to 21% + (JetBrains, P). +- **A fifth to a quarter of merged work is agent-authored.** 19.4% of public pull requests in August + 2026 carried an agent mark (Amplifying, P). One review vendor's data put fully AI-authored merged + pull requests at 27.6% in April 2026 (Greptile, vendor, P). +- **Much of it merges unreviewed.** In a sample of 268 merged agent pull requests in significant + repos, 56% had no human reviewer (Amplifying, P). +- **Trust is conditional.** 48% of developers trust AI output only when they can easily verify it + (Stack Overflow 2026, 30,903 responses, P). +- **Output is up; delivery is not.** Across about 22,000 developers, high-adoption periods showed + task throughput up 34%, bugs per developer up 54%, incidents per pull request up 243%, median + review time five times longer, and deployments per week down 11.7% (Faros, an analytics vendor; + correlational, P). The one randomised programme says it can no longer measure the effect cleanly + (METR, February 2026, P). + +### 2.4 Trends since early 2026 + +1. **Long, unattended runs.** Vendors describe agents working for hours or days. First-hand accounts + of overnight runs from ordinary solo builders are scarce. +2. **Verification is the binding constraint.** One vendor uses those words (Augment, P); the + telemetry above agrees; code review is the most acquired adjacent category. +3. **Seats became metered usage.** GitHub moved Copilot to token-metered credits on 1 June 2026, + saying flat pricing was "no longer sustainable" (P). Cursor's review product moved from a seat + price to a per-run price (S). +4. **Providers are absorbing the stack.** They ship their own control planes (Claude Managed Agents, + Codex cloud, GitHub Agent HQ hosting third-party agents) and are buying runtimes, package managers + and cloud development environments (S). +5. **Agents on CI.** GitHub Agentic Workflows went from technical preview in February to public + preview in June 2026 (P, S). +6. **Agents with write credentials in CI are the attack surface.** Three disclosed incidents in 2026 + share one root cause: an agent holding a write token read untrusted repository text as + instructions (Cloud Security Alliance research note, S). +7. **Churn.** Shut, sunset, sold or renamed within about a year: Charlie Labs, Vibe Kanban's company, + Crystal, Firebase Studio, Amazon Q Developer, Windsurf, Gemini CLI for subscribers, Graphite, + Gitar, Cursor, and Augment's Cosmos, sold to Harness two days before this report (P and S). + +### 2.5 Pricing and business models + +| Model | Who | What it means for an unattended factory | +|---|---|---| +| Flat subscription with rolling limits | Claude, Codex, Factory | Cheapest by far while it lasts. Limits are unpublished and changeable | +| Subscription that is a prepaid balance at API rates | Copilot, Amp, v0, Replit | Predictable only up to the balance | +| Per review or per run | Claude Code Review at about $15–25, Bugbot at about $1–1.50, Greptile overage | Scales with pushes | +| Bring your own key or plan | Most open projects; Tembo, Amp, Warp, Devin increasingly | The operator carries the model bill | +| Outcome-based | One product, per merged change | Rare | + +**The line that matters.** An individual working interactively stays inside $100 to $200 a month on +one flat plan. Anyone running agents unattended, in CI or in parallel is being moved onto metered +credits or API keys, where credible figures run from the low thousands a month upward. + +**The specific risk to this org.** Anthropic announced in May 2026 that programmatic use of Claude +Code (`claude -p`, the Agent SDK, GitHub Actions) would move off subscription limits onto a small +monthly credit and then API rates, and paused the change on its 15 June start date. No new date has +been given. Its guidance for automation shared across repos is an API key. Its monthly API credits +exclude runs started by its GitHub Action (S for the announcement and pause; P for the pricing page, +which says Claude Code "shares the same usage limits as the rest of your plan"). Google withdrew its +CLI from individual subscribers on about thirty days' notice (P). GitHub's charge for self-hosted +runner minutes is postponed, not withdrawn; public-repo minutes "will remain free" (P). + +## 3. Customer insights: the solo builder + +### 3.1 Segments and behaviour + +| Segment | Defined by | Fit for a GitHub-native factory | +|---|---|---| +| A. Non-technical founders on prompt-to-app builders | Cannot read the code; ship from a chat box | **None.** The largest group by head-count and excluded by construction | +| B. Technical indie hackers on terminal and editor agents | Review code, own the repo and the deployment | **Best.** The natural early adopter and the best-evidenced segment | +| C. Solo open-source maintainers | Receive agent output as well as produce it | Partial: their pain is inbound | +| D. One-person agencies | Sell delivery | Thin evidence | +| E. Portfolio builders | One person, many small products; attention is the limit | **Good**, and closest to this org. Thin evidence: a handful of loud accounts | + +How segments B and E work today: + +- **Two to five agents at once, with a ceiling near eight.** The limit people name is their own + attention, not tooling (several first-hand accounts, P). +- **The human moved up, not out.** Planning and acceptance are where the person now sits. One + developer puts about 80% of their time into thinking (P). +- **A second model as reviewer is common practice.** One builder runs a push hook that needs two + frontier models to approve and fails closed (P). +- **Spend sits in three bands:** up to $20 a month for most; $100 to $200 for serious builders; a + small tail spending far more through stacked subscriptions. +- **Workflows have a short shelf life.** Builders report abandoning whole setups when a model + release changes what works, and dropping their own orchestration when the agent gains sub-agents. + +### 3.2 Pain points, ranked + +| # | Pain | Evidence | What people do today | +|---|---|---|---| +| 1 | Review burden and low-quality output | Strong: two surveys, telemetry, several widely read essays | A second model, plan first, thin slices; or stop reading code | +| 2 | Usage caps, unpredictable cost, plan changes | Strong: about 30% hit limits; repeated large threads | Ration, switch tool, stack accounts, turn off overage | +| 3 | The gap from prototype to production | Strong for segment A: a paid rescue trade exists | Hire help, rewrite, give up | +| 4 | Passes tests and is wrong; tests edited or results misreported | Moderate: many concrete reports, no survey | Protect tests, check outside the model | +| 5 | Supervising many agents | Moderate to strong: consistent accounts, vendor telemetry | Cap at two to five, notifications | +| 6 | Leaked secrets, destructive actions, guardrails that are only advice | Strong on incidence, weaker as a felt pain | Sandboxes, scoped tokens; often nothing | +| 7 | Context lost across sessions | Moderate | Instruction files, task memory, fresh sessions | +| 8 | Lock-in and sudden policy change | Moderate and spiky | Use two to four tools at once | + +This org's own telemetry shows pains 1, 2, 4 and 5 at fleet scale (architecture review §2, §14). + +### 3.3 Unmet needs + +1. **A stopping rule they can trust when nobody is watching:** "done" checked by something the agent + cannot argue with. Strong evidence. +2. **A hard cap on spend.** People switch overage off after one surprise. Nobody was found asking for + cost per shipped change in those words; the need is implied. +3. **Review that scales past one person without becoming no review.** +4. **Guardrails without a platform team.** Solo builders assemble them from containers and scoped + tokens. Strong evidence that the need exists; no product named as meeting it. +5. **A record of what the agents did and why something is stuck.** Moderate; mostly inferred. +6. **Operating a portfolio of repos.** Thin evidence. + +### 3.4 How they decide, and what they pay + +- **Tools are added, not swapped.** 70% use two to four (Pragmatic Engineer, 906 responses, P). +- **Quality moves people about four times as often as price.** 25.1% switched for better output, + 6.4% for lower cost (Stack Overflow 2026, P). Caps and policy shocks are the acute triggers. +- **Decision criteria, by weight:** output quality (very high); usage headroom per dollar (high); + works with the subscription and agent they already have (high for this layer); control and + visibility (high among technical builders); low setup cost (high); open and self-hostable (medium + to high); price (medium, with a hard floor near $20); autonomy (medium and polarising). +- **Willingness to pay for the model:** $20 by default; $100 to $200 while it buys far more than its + API-rate value. +- **Willingness to pay for orchestration on top: low.** Vibe Kanban had thousands of daily users, + most on the free tier, and its company closed saying it could not find a business model (P). + Prices that exist at this layer are free or about $20 a seat. +- **Most indie products earn little.** Two analyses put median revenue near $150 a month (S), which + caps what the median builder can justify. + +### 3.5 Barriers to a factory, as opposed to one agent + +| Barrier | Strength | +|---|---| +| **No proof it beats one good agent.** Research shows multi-agent setups hurting sequential work; practitioners report two or three focused agents beating six to eight; the author of the best-known orchestrator abandoned reusable orchestration | **Strong: the main barrier** | +| The platform absorbs the feature | Strong | +| Cost multiplication and fear of runaway spend | Strong | +| Trust | Strong | +| Setup complexity | Moderate to strong | +| Too little designed work to feed it: design is the bottleneck once code is cheap | Moderate | +| Fear of breaching subscription terms | Moderate | + +### 3.6 Signals for and against + +**For.** About a dozen open projects created in eighteen months hold 6,000 to 100,000 stars each, +including a self-hosted control plane with budgets and approvals at about 100,000. The official +action for running Claude Code on GitHub's CI has about 9,500. Vendors are shipping scheduled agents +natively, which confirms the need. + +**Against.** Attention has not converted to revenue. The most-read essays of 2026 argue for less +autonomy. One detailed account of a solo, self-hosted factory moved off GitHub for isolation, CI +limits and outages. Nothing shows solo builders want this *GitHub-native* in particular. + +## 4. Competitive landscape + +### 4.1 How much of the loop each automates + +*auto*: runs on an event or schedule with no per-item human action. *trig*: the agent does the step +once a person, label or mention starts it. *human*: a person does or must approve it. A dash: not +offered. A question mark: not established. + +| Product | Intake | Plan | Build | CI repair | Review | Merge decision | Release | Operate | +|---|---|---|---|---|---|---|---|---| +| GitHub Copilot stack | trig | trig | trig | trig | auto | trig, preview | — | — | +| GitHub Agentic Workflows | auto | trig | trig | trig | trig | refused | trig | ? | +| Claude Code cloud and routines | auto on schedule | trig | trig | auto | auto | human | trig | trig | +| OpenAI Codex | trig | human | trig | trig | auto | — | — | — | +| Cursor | trig | auto, beta | trig | auto | auto | human | monitor only | trig | +| Cognition Devin | trig | auto | trig | auto | auto | human | — | trig | +| AWS Kiro | trig | auto | trig | auto | human | refused | — | — | +| Factory | trig | human | trig | trig | auto | ? | human, preview | trig | +| Warp Factories | trig | human | trig | trig | auto | refused | — | trig | +| Harness Cosmos | trig | ? | trig | auto | auto | human | ? | trig | +| Gitar | — | — | — | auto | auto | auto, opt-in | — | — | +| Gas Town | human | trig | auto | ? | ? | auto on gates | — | — | +| *This org's factory, as designed* | *auto* | *auto* | *auto* | *auto* | *auto, council* | *policy; human approves outcomes* | *staged, of the factory itself* | *not in the MVP* | + +Reading it: + +- **Build, CI repair and review are commodity.** Eight or more products do each. +- **The merge decision is the step every vendor hands back.** Kiro "never merges changes + automatically"; Warp: "The factory never merges"; Cosmos: a human "always owns and performs the + final merge"; Agentic Workflows refuses merges to the default branch (all P). +- **The pieces of an agent-approved merge exist on GitHub, switched off.** Copilot code review can + submit an approval that counts toward required reviews, in preview since 1 September 2026 and off + by default (P). Several review vendors document an opt-in auto-approve. +- **Release and operate are empty at solo price.** What exists is preview or enterprise. + +### 4.2 Positioning + +Two axes separate the field: how much of the loop runs unattended, and how much control the operator +keeps (declared state, the agent's own identity, a reviewer that is not the writer, loop and cost +bounds, audit, configuration in the operator's own Git and CI). + +- **High automation, high control: thinly occupied.** Warp Factories, Harness Cosmos and Factory's + preview come nearest. All stop at merge and are early-access, enterprise or contact-gated. +- **Moderate automation, high control: the toolkit band.** GitHub Agentic Workflows has the strongest + published control model: a read-only agent job, separate jobs that apply its requested effects, a + threat-detection job, and credit ceilings per run. It is open-licensed, a public preview, and a + toolkit the operator must assemble per repo. +- **High automation, lower control: trust the vendor.** Cursor has the widest coverage. Devin and the + Copilot stack follow. The same vendor writes, reviews and can approve. +- **Narrow, moving up: the verification layer.** Gitar, cubic and CodeRabbit own three columns and + are extending toward fix-and-merge. +- **Different game.** Prompt-to-app builders automate an idea into one hosted app. None takes + issues, reviews code or manages releases across existing repos. + +### 4.3 What is commodity, and what nobody offers + +**Several now offer it; building it is redundant:** + +- Issue-to-pull-request implementation in a sandbox. +- CI repair and review follow-up on the agent's own pull request. +- Automatic first-pass review, free on public repos. +- Event and schedule triggers. +- A separate agent identity. +- Requested effects with a separate applier (Agentic Workflows). + +**Nobody offers a solo builder:** + +1. A governed, unattended merge decision on the operator's own policy, with an approver that is not + the writer. +2. Release and operation at solo price. +3. A view across a portfolio of why each work item is waiting. +4. One versioned definition rolled out in stages across about a dozen public repos. +5. Admission of work against a flat subscription's remaining capacity. +6. Verification by reviewers from different vendors plus deterministic tools. +7. Holdout verification as software. StrongDM's version ships as three specification files. + +### 4.4 Threats and openings + +#### Threats to a small, open, GitHub-native factory + +- **GitHub absorbs the control plane.** Triage controls, automations, review with approval, an agent + merge and Agentic Workflows together cover intake through merge. Its automations are not available + on public repos today, with support described as coming. +- **Subscription terms.** §2.5. This is the largest single risk to the economics. +- **Free review tiers** remove a reason to build reviewers. +- **Churn.** Anything the factory integrates may not exist next quarter. +- **The name is taken.** Companies with hundreds of millions in funding sell "software factory". + +#### Openings: what vendors structurally will not do + +- **Take the merge decision on the operator's policy.** A vendor that merges owns the blame. An + operator can accept that risk for their own repos. +- **Verify with competitors' models.** A council mixing deterministic tools, tests and reviewers from + different vendors is against each model vendor's interest. +- **Minimise the operator's spend.** Vendors earn from metered usage. This opening is narrowing as + more products accept the user's own plan. +- **Serve one person on public repos with full governance.** Audit, service accounts and managed + review sit on team and enterprise tiers. +- **Keep everything in the operator's Git and CI,** with nothing always on. +- **Roll the automation itself out in stages across a portfolio.** Not documented by anyone. + +## 5. Strategic implications + +1. **The differentiated part is small, and it is the right part.** The market has commoditised the + workers and first-pass review. What it does not offer is the control plane around them: the merge + decision, the council, portfolio rollout, budget admission. That is the part ADR-0013 proposes to + build. The research supports the choice and narrows it. +2. **Adopt what is commodity.** Building a worker harness is competing with ten vendors and with an + open-licensed toolkit from the platform owner. The run contract should be shaped so that toolkit, + or a vendor's agent, can fill it. +3. **The council is the strongest opening and the least crowded.** No product documents reviewers + drawn from independent vendors plus deterministic tools. +4. **Plan the economics for metered capacity.** A design that works only while subscription + automation is tolerated has a single point of failure outside the operator's control. +5. **The customer is the technical solo builder and the portfolio builder.** The largest solo + segment cannot use this and should be named as out of scope. +6. **Prove it against one agent.** The strongest objection in the market is that orchestration does + not beat a single well-run agent. The factory's own Ledger can answer that and should be made to. +7. **Do not plan on selling the orchestration layer.** Attention is large; willingness to pay is low; + the platform owner is closing in; and one vendor's terms bar third parties from offering its + subscription login in their products. The evidence supports a factory built for this org, open + for others to read. + +## 6. Risks + +| Risk | Likelihood | Effect | Mitigation | +|---|---|---|---| +| Subscription automation is restricted or repriced | High: announced once, paused, not cancelled | Capacity cost rises several-fold | Engine independence; a costed metered fallback; hard spend caps | +| GitHub ships a governed agent merge and public-repo automations | High within a year | The control plane's distinct surface shrinks | Build on its primitives; keep the Kernel separable; adopt its toolkit where it fits | +| A factory does not beat one agent for one person | Unknown | The premise fails | Measure against a single-agent baseline before widening scope | +| An integrated product disappears | High | Rework | Contracts, not integrations; the registry of §16 of the review | +| An agent holding a write credential reads hostile text | Demonstrated three times in 2026 | Secret theft, poisoned releases | The run contract's credential boundary | + +## 7. Implications for the PRD + +Applied to the PRD in this revision: + +1. **§2 Why now:** the industry telemetry that matches this org's own. +2. **§1 Vision:** what the market does not offer, as the statement of what is distinctive. +3. **§3 Users:** the technical solo builder and portfolio builder as the user; non-technical founders + as a named non-user. +4. **§5 Constraints:** capacity terms can change on short notice (new C-11); C-3 records the paused + change to programmatic use. +5. **§6:** a requirement that the Factory can run on metered capacity under a hard cap; a requirement + to adopt commodity workers and reviewers through the contract. +6. **§9 Success metrics:** a comparison against a single-agent baseline. +7. **§10 Scope:** a build-versus-adopt table. +8. **§11 Open questions:** whether to adopt GitHub Agentic Workflows as the worker half; the metered + fallback budget. + +Left for the maintainer, and not changed: + +- **The non-goal "not a product for other organisations".** The evidence supports keeping it (§5, + item 7). It could be restated as "not a commercial product; open for others to read and reuse". +- **Release and operation.** The widest gap in the market and outside the MVP. Whether it becomes + the next scope after exit is a strategy decision. + +## 8. What could not be established + +- A size for the orchestration or "software factory" category. +- The number of solo builders running coding agents, or its growth. +- Any survey data specific to solo builders; anything first-hand from Reddit. +- The price at which a solo builder leaves an orchestration product. +- Demand for GitHub-native operation specifically, or for cost per shipped change as a metric. +- Whether unattended overnight runs work for ordinary builders. +- Independent quality evidence for any agent's merged output. +- Real monthly spend for a solo operator on any commercial product. +- Numeric limits on the flat plans of Anthropic, OpenAI, Cognition, Cursor and Google. +- Whether a third-party bot's approval satisfies GitHub's required-review rule. +- Revenue for Codex, Copilot, Factory and 8090; several vendors' pricing pages. +- The 2026 editions of DORA and Octoverse, which appear unpublished. + +## 9. Sources + +Market and adoption + +- JetBrains, AI coding agent adoption 2026: +- Stack Overflow Developer Survey 2026: +- Faros AI, "The Acceleration Whiplash": +- Amplifying, State of the Coding Agent Market: +- Greptile, "Rise of the overnight agents": +- METR, uplift update: +- Pragmatic Engineer, AI tooling 2026: +- StrongDM Software Factory: +- Augment, "What is a software factory": +- TechCrunch on Warp Factories: +- Cloud Security Alliance research note on coding-agent incidents: + +Pricing and terms + +- GitHub, Copilot usage-based billing: +- Claude pricing: +- Anthropic, using the Agent SDK with a Claude plan: +- Hacker News, "We're pausing the Agent SDK credit change (Anthropic)": +- Google, transitioning Gemini CLI: +- GitHub Actions runner pricing: + +Competitors + +- GitHub, Copilot code review can approve pull requests: +- GitHub, about Copilot automations: +- GitHub Agentic Workflows: , +- Claude Code routines and projects: , +- Kiro autonomous agent: +- Factory: , +- Gitar: + +Customer + +- Vibe Kanban shutdown: +- Steve Yegge, "The shape of things to come": +- Maggie Appleton on Gas Town: +- Manuel Schipper, parallel coding agents: +- Jake Saunders, a self-hosted factory: +- Google Research, scaling agent systems: +- Open projects' star counts: repository pages, read 2026-10-10.