Skip to content

docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed) - #2225

Closed
don-petry wants to merge 32 commits into
mainfrom
claude/agentic-factory-architecture-review-3cf4c4
Closed

don-petry wants to merge 32 commits into
mainfrom
claude/agentic-factory-architecture-review-3cf4c4

Conversation

@don-petry

@don-petry don-petry commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

What this is

Four documents that treat the current agentic factory as a proof of concept and define the production MVP. Documentation only: no workflow, script or prompt changes.

Document What it holds
docs/prd/agentic-software-factory.md The requirements: 73 functional requirements, 14 non-functional targets with POC baselines, 11 constraints, a build-versus-adopt table, exit criteria from POC to production
docs/architecture/factory-mvp-architecture-review.md The evidence and the reasoning: current-state findings, root causes, options analysis, comparison with other systems, an adversarial review, the convergence hypothesis, reviewer surfaces, where a Lens should run, and how it is built
docs/architecture/adr/0013-factory-mvp-reference-architecture.md The proposed decision, in the ADR-0000 form, with checkable boundaries. Status: proposed
docs/research/market-agentic-software-factories-research-2026-10-10.md Market research: what the market offers, the solo builder as a customer, a competitive teardown, and what it means for the PRD

Why

The POC proved every function of the factory and has stopped converging. Measured on 2026-10-10 at commit aef61164:

  • At least 44,968 workflow runs in seven days in this repo for 30 merged pull requests.
  • Merges per week flat since August while median time to merge rose from about 1 hour to 12–22 hours.
  • No open pull request in the org holds an approval; about a third of recent merges had no standing approval.
  • Eight fixes moved a failure somewhere else instead of removing it (review §2.4).

Seven structural causes explain the eleven recurring failure classes (review §4).

What ADR-0013 proposes

  1. One self-contained repository, new, with decision logic that does no I/O.
  2. A declared lifecycle and one state record per work item.
  3. Level-triggered admission in one place, with three brakes outside the kernel.
  4. One run contract; a job is a credential boundary; the model requests effects and a separate job applies them.
  5. Agents have their own identity; the owner account is the human's alone.
  6. One release, one path, promoted on evidence.
  7. An approval is bound to content.
  8. Verification binds: a required, hermetic check.
  9. One fact, one place.
  10. Migration: built greenfield, cut over as a strangler through this repo's channel tags, with a budget and a stopping rule set before anything runs.
  11. Two mechanisms are provisional, with thresholds: the git-backed store and per-event wakes.
  12. Lenses report through one contract; convergence is decided, not discussed.

It supersedes nothing on acceptance. It names nine follow-on ADRs, each landing with the migration step that makes it true.

A note on the ADR number

This record was opened as ADR-0012. While the PR was in review, ADR-0012 was assigned on main to the maintainer comment gate's exemption set, so the factory MVP record is now ADR-0013. The ADR's Context says how the two relate.

What it does not claim

The adversarial review (review §14) found that this architecture removes waste under the machine limits and does not raise the two ceilings that bind: the maintainer's attention, and how often a change is right the first time. Both documents say so.

How to read it

  • Ten minutes: review §1 (summary), then ADR-0013.
  • To challenge it: review §14 (the ceiling, where it fails first) and §13 (what other systems chose differently).
  • To decide scope: PRD §11 (open questions) and §10.3 (exit criteria).

Decisions this needs from the maintainer

Listed in PRD §11. The ones that shape everything else:

  • Do agents move off the owner account (OQ-2)?
  • Is the POC frozen for features while it is replaced, and what may the migration cost (OQ-9)? ADR-0013 is not in force until a dated limit, a POC defect policy and a kill criterion are written.
  • Which outside reviewers are removed now (OQ-13)?

Decided so far

  • Where it is built (2026-10-10): in a new repository, cut over through this repo's existing channel tags one intent and one ring at a time. Not in place, and not a rename or one-step swap, because GitHub Actions does not redirect reusable-workflow references. Review §18; ADR-0013 decisions 1 and 10.
  • Sourcing (2026-10-10): the market research found the workers are a commodity and the line around them is not, so the PRD now says the Factory builds the control plane and adopts workers and reviewers through its contracts. Market research §4.3; PRD §10.4, FR-72.
  • Third-party reviewers (2026-10-10): integrated through a structured contract; comment bodies gate nothing. Review §15.5 and §16; ADR-0013 decision 12.

Verification

  • markdownlint-cli2 with the repo config: 0 issues across the three files.
  • Citations to code are path:line at aef61164; the highest-consequence ones were re-checked by hand, and the branch ruleset was read live.
  • Platform claims were read from vendor documentation on 2026-10-10; review §11 and §16.7 list what could not be verified.

Not in this PR

Fifteen defects found in passing (review §12) are listed but not fixed here.

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

Problem

See the linked issue.

Risk

Low — documentation/configuration only; no runtime code paths change.

Test plan

No test files were added or updated. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the existing CI (bats + lint) guards the change.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

n/a — no runtime effect; documentation/configuration only.

Treats the current agentic factory as a proof of concept and defines the
production MVP.

- docs/prd/agentic-software-factory.md: requirements (FR-1..FR-71),
  NFR targets with POC baselines, constraints, exit criteria.
- docs/architecture/factory-mvp-architecture-review.md: evidence and
  current-state findings, root causes, options analysis, comparison with
  other systems, adversarial review and ceiling, the convergence
  hypothesis, reviewer surfaces and a starting council, where a Lens runs.
- docs/architecture/adr/0012-factory-mvp-reference-architecture.md:
  the proposed decision, with checkable boundaries. Status: proposed.

Documentation only. No workflow, script or prompt changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 10, 2026 21:31
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

This comment has been minimized.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces two major documentation files: a Product Requirement Document (PRD) defining the production MVP requirements for the Agentic Software Factory, and an Architectural Decision Record (ADR-0012) outlining the reference architecture for a rebuilt core on GitHub Actions. The review feedback highlights minor formatting and consistency issues in the PRD, including out-of-order success metrics and requirement numbering, as well as missing [ASSUMPTION] tags in the main text for several indexed assumptions.

Comment thread docs/prd/agentic-software-factory.md Resolved
Comment thread docs/prd/agentic-software-factory.md Resolved
Comment thread docs/prd/agentic-software-factory.md Resolved
The maintainer chose the hybrid on 2026-10-10: build the factory
greenfield in a repository of its own and cut over as a strangler at
this repo's existing channel tags, one intent and one ring at a time.

- ADR-0012 decision 1: a self-contained repository instead of a tree in
  this repo. Decision 10: forwarders, tag-move cutover and rollback, no
  rename while callers reference this repo, consumers re-pinned last,
  and a parity bound from a read-only run on live traffic.
- Review section 18: the two options, the two platform facts that decide
  the shape, the hybrid, its costs and conditions.
- PRD: constraint C-4 and open question 5 updated; glossary clarifies
  "Hosting repo" during the migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codeant-ai

codeant-ai Bot commented Oct 10, 2026

Copy link
Copy Markdown

CodeAnt Nitpicks

3 code suggestions

1. If GitHub accepts an Effect and the process crashes before recording success, the local intent key cannot identify the remote action; retrying may duplicate comments or reviews.

Incomplete implementation · docs/architecture/adr/0012-factory-mvp-reference-architecture.md:169-171


2. permissions: {} does not block the Actions cache service; untrusted verify code can poison a default-branch cache that later jobs restore.

Security · docs/architecture/factory-mvp-architecture-review.md:1712


3. Planning roles remain on their current runtime with the write token in the model, so this unqualified claim that every Role uses the split contract is false.

Comment mismatch · docs/architecture/factory-mvp-architecture-review.md:36-37

@codeant-ai

This comment has been minimized.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-10T22:58:06Z.

@donpetry-bot

Copy link
Copy Markdown
Contributor

No description provided.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • docs/prd/agentic-software-factory.md:653 — not applied
  • docs/prd/agentic-software-factory.md:517 — not applied
  • docs/prd/agentic-software-factory.md — applied
  • docs/architecture/adr/0012-factory-mvp-reference-architecture.md:163 — applied
  • docs/architecture/adr/0012-factory-mvp-reference-architecture.md:330 — applied
  • docs/architecture/factory-mvp-architecture-review.md:1279 — applied

The unaddressed items above still need work.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-10T23:10:45Z.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Nitpick 3 fixed: docs/architecture/factory-mvp-architecture-review.md summary now says 'Every role in scope for the MVP runs under one contract', so it no longer overclaims for the planning roles that §Gaps defers. Nitpick 1 was already handled in ADR-0012 §4 ('Every Effect type declares a recovery key… an intent record is committed before the Effect is applied'). Nitpick 2 was already handled: the review's R6 row now requires run and verify to run without permissions or cache writes, and §12 notes the default-branch cache-poisoning path.

@don-petry

This comment has been minimized.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Acknowledged: the Codex usage-limit notice carries no finding.

@don-petry

This comment has been minimized.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

@don-petry
don-petry enabled auto-merge October 10, 2026 22:22
@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

donpetry-bot and others added 3 commits October 10, 2026 18:25
…nst it

Adds a market research report (BMAD market-research workflow, solo
builders as the priority segment) and revises the PRD in light of it.

Research: market definition and size, adoption and pricing trends, the
solo builder as a customer, a competitive teardown by how much of the
idea-to-operate loop each product automates, threats and openings.

PRD:
- Vision: what the market does not offer, as the statement of what the
  Factory builds; it adopts the workers.
- Users: the technical solo builder running a portfolio; non-technical
  founders as a named non-user.
- Constraint C-11: capacity terms can change on short notice.
- FR-72: workers and reviewers are adopted through the contract.
- FR-73: the Factory can run on metered capacity under a hard cap.
- SM-13: the Factory measured against one agent.
- Section 10.4: build versus adopt.
- Open questions 14 to 17.

Documentation only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ure-review-3cf4c4' into claude/agentic-factory-architecture-review-3cf4c4

# Conflicts:
#	docs/prd/agentic-software-factory.md
ADR-0012 was assigned on main to the maintainer comment gate's exemption
set while this record was in review. ADR-0000 makes a number permanent
once assigned, so the factory MVP reference architecture takes 0013.

- Rename the file and update every reference in the PRD, the review and
  the market research.
- The ADR records the renumbering and its relationship to ADR-0012,
  which governs the POC's comment gate until the merge-integrity
  follow-on supersedes it.
- Review section 9 gains a row for ADR-0012.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry don-petry changed the title docs: factory MVP — PRD, architecture review and ADR-0012 (proposed) docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed) Oct 10, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

Neutral overview — no specific code defects to address. The assessment confirms the PR is safe to merge as documentation-only changes.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning
## Summary
**Bot:** `codeant-ai[bot]`
**Issues addressed:** 0 (non-actionable)
**Comment dispositioned:** 
- **"CodeAnt PR Risk: Low Risk"** (IC_kwDOR9SdIs8AAAABa7w3Kw): Neutral overview with no specific code defects or actionable findings. The comment states the PR is safe to merge as documentation-only changes. Posted `informational` disposition to clear the maintainer-comment gate.
**Files changed:** None (documentation review only)
**Notes:**
- The comment was edited after a prior disposition, making the stale disposition outdated. Posted a fresh `informational` disposition reply per instructions.
- No CI failures or CHANGES_REQUESTED reviews — no Tier 1 blockers present.
- The separate CodeAnt Nitpicks comment exists on the PR with 3 code suggestions, but the assigned task focuses only on the "Low Risk" summary comment.

Facts carry completeness flags and an incomplete read never converges.
apply re-reads the item before any Effect. A Run starts only from a
Decision a pass recorded. Failed Runs are classed as item, capacity or
environment faults; only item faults are attempts, and environment
faults pause Admission once. Every pass records what it read and
decided. The required check uses fixed seeds. verify is a filter and
the consumer's own checks stay the verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
donpetry-bot and others added 7 commits October 10, 2026 21:33
An independent cross-read found three design holes and about thirty
stale statements.

- The first stable Release was unreachable: promotion needed evidence
  from a next ring that had no member. The factory's own repo is now
  served by next, and stable serves consumers.
- A monthly spend cap survived after money stopped gating. It now
  applies on a metered credential and is read from the provider; on the
  subscription the brake is the Run ceiling.
- Cutover still called a mechanism the poller removed. It is now pause,
  then repoint, with the gap accepted.

Also: the Role table and examples show the shadow start; the PRD's
FR-2, FR-32, FR-33, FR-50, FR-51 and glossary are brought into line;
wake and ingress remnants removed from the ADR; the review's summary
and section 18.5 are annotated where they state replaced positions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Work items have a class; an outside item gets no Run until the
maintainer admits it. The approver is never the author or last pusher,
so the machine user never pushes. Model Lenses run only after the
deterministic ones pass. Attempts count since the last human action.
An empty diff never converges. The reviewer resolves third-party
threads it does not adopt, so required thread resolution stays in
force. The maintainer alone is code owner for the factory's own rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plain comment steers and a review from a person with write access
gates; a thread from anyone else is handled like a third-party
reviewer's. The control vocabulary is written down as one GitHub action
per control. The factory reaches the maintainer by assigning or
requesting review. Consumers get a readiness check. Apps are installed
across the org with tokens minted per repo. Shadow Roles review as well
as adjudicate. The review is marked as history, and the design document
says which of its sections are rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Convergence was the absence of objection. It now also needs evidence:
where a work item has an acceptance contract, every criterion must have
passing evidence on the content to be merged. The contract is drafted
before the work by qa-lead, using the vendored BMAD test-architecture
workflows for test design, ATDD and the first phase of trace; it is
approved by the maintainer, which is what arming the work means; and
the implementing Role cannot change it. It takes effect with the fix
and implement intents, loosely at first. The factory's own invariants
and simulator seeds join the paths only the maintainer may approve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a person with write access resets an agent's attempts. The
acceptance contract is the first commit of the work's one pull request,
so failing tests never reach the default branch. Unadopted Findings
later proved right are counted. The maintainer is notified once per
item per reason, and when more than ten items await them only work that
unblocks a merge is admitted. Issues come under factory control by a
label. Held-out eval sets live under a different owner. Shadow Roles
have a budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Everything the factory produces is public, so an injected model can
publish the model credential through a log with no network access.
Three layers answer that: no shell or network tool for a Run that reads
untrusted text, the model CLI's sandbox pending a trial, and a model
credential that is the factory's alone.

Run records count by workflow, event and repo, never by name, so a
fork cannot forge them. The control-room feed carries no free text from
outside. An outside item's admission lapses on the next push. The
external timer holds no credential. An approval is bound to the
admitted commit. Every write by a factory identity is reconciled with
a recorded Effect. A credential register is added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the design on one page; a ranked order for the platform trials,
with three new ones; the critical path to the first approval, which
puts an eval set for the review intent on it; a table of requirements
not yet designed; a comparison of the POC, the ADR's first draft and
this design; and the strongest case for not building it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Don Petry and others added 2 commits October 11, 2026 09:30
…sign

The factory's own author Role moves ahead of the control room and the
shadow Roles. A weekly scorecard compares the POC and the factory. The
factory's records last 90 days and the control room may keep its own
without limit. A killed Run is costed by estimate; resumable Runs and
graceful degradation are deferred. The fallback for the model
credential is a metered API key.

A reviewing Role declares a rubric, which is a floor and grows from
what it missed. A hosted relay turns the Controller App's webhooks and
a timer into nudges, so no consumer repo carries a factory file. The
build practice before the factory writes its own code is set out.

The POC fixes the review recommended are tracked in the epic issue
#2276 and not in this pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Research on 2026-10-11 into state-machine libraries, policy engines,
specification tools, merge orchestrators and workflow engines found no
framework that fits a pure function over facts read afresh on every
pass. The design document records the verdict per part of the Kernel,
what would change it, and what it implies for the language: Python with
the standard library only at run time is recommended, with a
property-testing library at test time and Quint held in reserve for a
formal model. The maintainer has not yet ruled on the language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T15:42:07Z.

…context

The merge queue is in use on .github-private, which settles its
availability. Slice 0 gains two things on the maintainer's instruction:
a small proof of concept of the Kernel, schemas and validators, built
first and locally; and context engineering, meaning AGENTS.md files,
standards, a decision log that starts again at number one, and the
lessons of the POC, each tied to a check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T15:50:49Z.

The proof of concept answers its questions yes, and its simulator found
two errors in the design, both corrected here: whose turn it is on an
open thread cannot be read from the last Run on the current content,
because the author's answer is usually new content; and a Run that is
admitted but not yet started must count as holding the item. It also
adds one rule: a Run asks the Kernel again as it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T15:59:48Z.

Don Petry and others added 3 commits October 11, 2026 10:04
A dependency-free TypeScript port reproduces all 10,785 decisions
recorded from the Python Kernel. The comparison is added to the design
document. Neither language is disqualified; the recorded decisions make
the choice reversible. The recommendation stays Python, for the purity
check and the runner's default interpreter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruled by the maintainer on 2026-10-11 after the trial of both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three identities are set separately in a session and all must be the
maintainer's. Nobody can approve their own pull request, so through
slices 0 and 1 the ruleset requires the check and the merge queue and
no approving review; required review starts when the factory's reviewer
serves the repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T16:17:58Z.

@don-petry

Copy link
Copy Markdown
Collaborator Author

These six documents now live in petry-projects/software-factory, the repo the Factory is built in.

They were brought in from this branch at commit 7d7d04f by the pull request "Bring the six design documents in from .github-private" (combsmith/software-factory#1), which is open and awaiting merge. Paths are unchanged, and each document gained one line saying it was first written here.

Closing this unmerged. The branch claude/agentic-factory-architecture-review-3cf4c4 is kept as the record of how the design was reached. Further changes to the design are made in the new repo.

@don-petry don-petry closed this Oct 11, 2026
don-petry added a commit that referenced this pull request Oct 11, 2026
…ence and never fakes it (#2295)

* docs(adr): propose ADR-0014 — the canary gate signals incomplete evidence and never fakes it

Records, as one proposed decision, what the canary-rollout gate does when a
read it depends on is capped: one completeness rule for run lists, a gating
window that is completed or held, unknown baseline days that are not zero,
the clamp minimum for an unknown baseline (maintainer ruling 2026-10-11),
an indeterminate triage when nothing was compared, role eligibility by
ingress_job, and an ingress read budget that counts uncached reads once job
reads persist across sweeps.

Numbered 0014 because the open draft PR #2225 holds 0013.

Refs combsmith/.github#1244

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(adr): apply review wording suggestions to ADR-0014

Seven wording fixes from the Gemini review: "jobs API read", "exceed", "run
ID", two commas, and the #548 reference phrasing. No change of meaning.

Refs combsmith/.github#1244

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Don Petry <{}+donpetry-bot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants