Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/approve-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
ITEM: ${{ inputs.item }}
PASS: ${{ inputs.pass }}
APPROVER_TOKEN: ${{ secrets.APPROVER_TOKEN }}
timeout-minutes: 4
run: python3 -m harness.approve_step
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand All @@ -44,6 +45,7 @@ jobs:
if: always()
env:
NAME: ${{ inputs.name }}
timeout-minutes: 2
run: python3 -m harness.outcome
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
env:
GITLEAKS_VERSION: "8.30.1"
GITLEAKS_CHECKSUM: "551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb"
timeout-minutes: 3
run: |
tarball="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/${tarball}"
Expand All @@ -44,6 +45,7 @@ jobs:
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
sudo mv /tmp/gitleaks /usr/local/bin
- name: Run gitleaks
timeout-minutes: 5
run: |
rm -f .gitleaksignore
gitleaks detect --source . --config base/.gitleaks.toml --gitleaks-ignore-path base/.gitleaksignore --ignore-gitleaks-allow --log-opts="--full-history --all --diff-filter=tuxdb --text" --redact --verbose --exit-code 1
5 changes: 5 additions & 0 deletions .github/workflows/eval.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,23 +27,27 @@ jobs:
with:
persist-credentials: false
- run: python3 -m harness.eval.plan
timeout-minutes: 1
env:
RELEASE: ${{ inputs.release }}
- run: python3 -m harness.eval.fetch
timeout-minutes: 3
env:
HELDOUT_TOKEN: ${{ secrets.HELDOUT_TOKEN }}
HELDOUT_OWNER: ${{ secrets.HELDOUT_OWNER }}
HELDOUT_REPO: ${{ secrets.HELDOUT_REPO }}
- env:
CLI_VERSION: "2.1.306"
CLI_CHECKSUM: "520bc6c1f40af466bd03a278b5ae1af1a2938bc7de5b9e5cb15526da20d71cfe"
timeout-minutes: 3
run: |
url="https://downloads.claude.ai/claude-code-releases/${CLI_VERSION}/linux-x64/claude"
curl -sSfL --proto '=https' --proto-redir '=https' -o /tmp/claude "${url}"
echo "${CLI_CHECKSUM} /tmp/claude" | sha256sum -c
chmod 0755 /tmp/claude
sudo mv /tmp/claude /usr/local/bin/claude
- run: python3 -m harness.eval.run
timeout-minutes: 110
env:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
CLAUDE_CLI: /usr/local/bin/claude
Expand All @@ -54,3 +58,4 @@ jobs:
if-no-files-found: error
retention-days: 90
- run: python3 -m harness.eval.judge
timeout-minutes: 1
2 changes: 2 additions & 0 deletions .github/workflows/pass.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ jobs:
- name: Read and decide
env:
CONTROLLER_APP_KEY: ${{ secrets.CONTROLLER_APP_KEY }}
timeout-minutes: 6
run: python3 -m controller
- name: Upload the pass record
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand All @@ -33,6 +34,7 @@ jobs:
env:
CONTROLLER_APP_KEY: ${{ secrets.CONTROLLER_APP_KEY }}
WORKFLOW_TOKEN: ${{ github.token }}
timeout-minutes: 6
run: python3 -m adapters.effects
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
with:
persist-credentials: false
- run: python3 -m release.move
timeout-minutes: 10
env:
RELEASE_COMMAND: ${{ inputs.command || 'cut' }}
RELEASE_VERSION: ${{ inputs.version }}
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/run-jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,13 @@ jobs:
persist-credentials: false
- name: Hand on the commit it checked out, which every later job checks out
id: commit
timeout-minutes: 1
run: python3 -m harness.steps
- name: Install the CLI at the Engine's pin
env:
CLI_VERSION: "2.1.306"
CLI_CHECKSUM: "520bc6c1f40af466bd03a278b5ae1af1a2938bc7de5b9e5cb15526da20d71cfe"
timeout-minutes: 3
run: |
url="https://downloads.claude.ai/claude-code-releases/${CLI_VERSION}/linux-x64/claude"
curl -sSfL --proto '=https' --proto-redir '=https' -o /tmp/claude "${url}"
Expand All @@ -53,12 +55,14 @@ jobs:
ITEM: ${{ inputs.item }}
PASS: ${{ inputs.pass }}
READER_APP_KEY: ${{ secrets.READER_APP_KEY }}
timeout-minutes: 3
run: python3 -m harness.read
- name: Run the Engine
id: engine
env:
CLAUDE_CLI: /usr/local/bin/claude
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
timeout-minutes: 20
run: python3 -m harness.run
- name: Upload the Result
id: result
Expand Down Expand Up @@ -94,6 +98,7 @@ jobs:
- name: Verify the Result
env:
RUN_DIGEST: ${{ needs.run.outputs.digest }}
timeout-minutes: 5
run: python3 -m harness.verify
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
Expand Down Expand Up @@ -136,6 +141,7 @@ jobs:
PASS: ${{ inputs.pass }}
RUN_DIGEST: ${{ needs.run.outputs.digest }}
PR_REVIEW_APP_KEY: ${{ secrets.PR_REVIEW_APP_KEY }}
timeout-minutes: 4
run: python3 -m harness.apply_step
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand All @@ -147,6 +153,7 @@ jobs:
if: always()
env:
NAME: ${{ inputs.name }}
timeout-minutes: 2
run: python3 -m harness.outcome
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The required check, `factory / verify`: it passes only when `make verify` and every shard of the mutation check pass on the commit, with the history tools/check_decisions.py compares it with.
# `checks` runs `make verify`, its simulators on 300 seeds. On 2026-10-21 the runs that finished took 11 to 14.5 minutes, and six runs in three pull requests were cut at its 15-minute timeout with the tests still running, so its timeout is 25 minutes. Raise it again before a run nears its timeout.
# `checks` runs `make verify`, its simulators on 300 seeds. On 2026-10-21 the runs that finished took 11 to 14.5 minutes, and six runs in three pull requests were cut at its 15-minute timeout with the tests still running, so its timeout was raised to 25 minutes; by 2026-10-29 a run took up to 23, so the job has 28 and its step 26, two kept for the checkout. Raise both before a run nears the step's.
# About 1,000 mutants, in eight shards of about 126, each kill confirmed by two runs of one test. With six shards of 168 the legs ran 14 to 25 minutes against a 25-minute timeout, and legs were cut at 25:00 in three pull requests, once 22 ms after the last kill had printed. Every mutant names the tests that kill it, which run first; the whole suite, its slow modules last, runs only when they pass ("Mutation run: every mutant names its tests, the slow suites run last, and a builder may break a few", issue 217). Add a shard before a leg nears its timeout.
# `verify` checks out the base, never the head, and runs the base's tools/needs_passed.py, so a change cannot make its own verdict pass: decision 0023.
# `commits` checks out the base too and runs the base's tools/commits_are_the_maintainers.py, which reads every commit the pull request carries and refuses unless each names the Maintainer ("Identity: every commit a pull request carries is the Maintainer's, checked outside the shell", issue 204).
Expand All @@ -11,7 +11,7 @@ permissions: {}
jobs:
checks:
runs-on: ubuntu-24.04
timeout-minutes: 25
timeout-minutes: 28
permissions:
contents: read
steps:
Expand All @@ -20,6 +20,7 @@ jobs:
persist-credentials: false
fetch-depth: 0
- run: make verify
timeout-minutes: 26
mutants:
runs-on: ubuntu-24.04
timeout-minutes: 30
Expand All @@ -46,6 +47,7 @@ jobs:
persist-credentials: false
fetch-depth: 0
- run: python3 -m tools.mutants
timeout-minutes: 27
commits:
runs-on: ubuntu-24.04
timeout-minutes: 10
Expand All @@ -62,6 +64,7 @@ jobs:
fetch-depth: 0
ref: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
- run: python3 -m tools.commits_are_the_maintainers
timeout-minutes: 7
verify:
name: verify
needs:
Expand All @@ -81,3 +84,4 @@ jobs:
persist-credentials: false
ref: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
- run: python3 -m tools.needs_passed
timeout-minutes: 5
21 changes: 16 additions & 5 deletions docs/architecture/software-factory-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,12 @@ GitHub's GraphQL gives it in capitals (`SUCCESS`, `IN_PROGRESS`); the port lower
nothing else. The configuration's `check_conclusions` says what each word means for a required
check. `success` alone is passing. `failure` and `timed_out` are failing: the content goes back to
the author, because a hung test usually belongs to the content and attempts bound the fixes (L-39).
GitHub concludes a job it stops at its own `timeout-minutes` `cancelled`, not `timed_out`, and its
check run with it (P-42). So each job the Factory writes stops its own work first: each `run:` step
has a `timeout-minutes` of its own, the steps' together two minutes fewer than the job's, kept for
its checkout and other steps, and a step stopped at its limit concludes `failure`, its job and check
run with it (`tools/check_workflows.py`, issue 388). A Lens whose work runs too long is then a
failing check, not one with no verdict, while its other steps keep within those two minutes.
`queued`, `in_progress`, `pending`, `waiting`, `requested` and `expected`, and a required check that
has not reported at all, are pending: a normal wait that ends on its own, so the item waits in
`verifying`. `neutral`, `skipped`, `cancelled`, `stale`, `action_required`, `startup_failure` and
Expand Down Expand Up @@ -911,9 +917,14 @@ cost is not known exactly; its duration is the estimate.
A pass reads `outcome.json` only for Runs that did not succeed, which are few. A killed Run has no
outcome file; one that ran, its conclusion `failure` or `timed_out`, is read as `item`, an attempt,
whatever other Runs are failing as (decision 0049, issue 286): the facts give no window to judge an
outage by, and the automatic pause stops admission on the Runs that did write `environment`. That a
job killed at its timeout concludes `timed_out` waits on P-18. A Run `cancelled` with no outcome is
no attempt: it is read as `superseded`, and so is a job that never started, `startup_failure`, which
outage by, and the automatic pause stops admission on the Runs that did write `environment`. A job
GitHub stops at its own `timeout-minutes` concludes `cancelled`, not `timed_out` (P-42), so the
Run's jobs stop their own work under a step limit, the steps' together two minutes below the job's:
a step stopped at its limit concludes `failure`, and its job with it, and a later `if: always()` step
still runs (`trials/P-42.step-timeout.json`). A Run past its limit is no attempt: `apply` still runs,
and leaves the outcome `environment`, `run_failed`, for any run job that did not succeed, so the
automatic pause and the Run ceiling bound it, not the attempts (issue 433). A Run `cancelled`
with no outcome is no attempt: it is read as `superseded`, and so is a job that never started, `startup_failure`, which
the records port reads as `cancelled`. Its concurrency group cancels a Run that waits behind another
as a newer one queues (P-44); a person's cancel of a waiting Run reads the same, and then the Run ceiling, not
the attempts, bounds it. GitHub lists a Run waiting for its group as `pending`, which the records
Expand Down Expand Up @@ -1829,7 +1840,7 @@ A sweep on 2026-10-10 ran the Lifecycle against awkward inputs. Each case below
| A dependency update | Its class is `dependency`, and the bot that opened it pushed its head. Its required Lenses are deterministic ones, unless it changes a protected path or a Lens's input, and a person's thread, or a model Lens's Finding already raised, still gates it. Once the author Role or a person pushes to it, it needs a model review like any change (§4.2). On a Consumer the POC's automerge serves it until that Intent is cut over. In the Factory's own repo nothing is installed, so every update changes `.github/`: every model Lens reviews its head, and the machine user approves it as any item (I-45) |
| The approver is the author or the last pusher | GitHub rejects or discards the approval. The Kernel reads `author` and `last_pusher`, attempts no approval, and holds the item with that reason (I-26). The machine user therefore never pushes (§5.1). The Hold is a reading: after a conflict, the Factory's fix Run pushes, the Role's App becomes the last pusher, and the Hold ends with no person, since GitHub would then count the approval (§4.1) |
| A branch pushed every few minutes | Deterministic Lenses first; a model review starts only on content they have passed (I-27) |
| A required check that concludes `neutral`, `skipped`, `cancelled` or `stale`, or with a word the Kernel does not know | It gave no verdict, so it has not passed. The Kernel defers, reason `check_unreadable`, admits no Run, and shows the item as needing the Maintainer until the check reports again (I-13, §3.3). A `timed_out` check is a failure, and goes back to the author. A failure beside it is read first, so `fail-fast`'s cancelled siblings and a skipped dependent job do not hold an ordinary failure |
| A required check that concludes `neutral`, `skipped`, `cancelled` or `stale`, or with a word the Kernel does not know | It gave no verdict, so it has not passed. The Kernel defers, reason `check_unreadable`, admits no Run, and shows the item as needing the Maintainer until the check reports again (I-13, §3.3). A `timed_out` check is a failure, and goes back to the author. A job GitHub stops at its own timeout concludes `cancelled` (P-42), so each job the Factory writes stops its own work under a step limit and concludes `failure` (§4.2). A failure beside it is read first, so `fail-fast`'s cancelled siblings and a skipped dependent job do not hold an ordinary failure |
| A check under a required Lens's check name from another App or another workflow file, or two from the Lens's own App and file that disagree | The first leaves the Lens unreadable, reason `check_unreadable` with the evidence `<lens>:from_app:<id>` or `<lens>:from_workflow:<path>`, for the Maintainer to see, whatever any Lens's own check says, a failure among them; it moves nothing either. A check run cannot be deleted, so it stays on that head for good, and only a push clears it. Of the second, every one is read: a failure fails the Lens, and it passes only when every one passes (§3.3) |
| A Hold released, or a pull request reopened, after attempts ran out | Attempts count since the last action by a person with write access, which grants a fresh allowance. No agent's action does, and nor does a stranger's comment: on a public repo that would let anyone keep an agent running (I-28) |
| An approval that does not land | The approval is a Run with no Engine. A failed one whose fault is the item's is an attempt, and at `max_attempts` the item is held. One that fails for any other fault, such as an expired token, counts toward the Run ceiling (§3.10) |
Expand Down Expand Up @@ -2152,7 +2163,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is
| P-15 | Do the retained Advisory reviewers review a pull request authored by an App? | Each one does, or is not counted as a Lens on such pull requests |
| P-16 | How long are run records, logs and artifacts kept in a public repo? | 90 days |
| P-17 | Is a run's name set when the run is created, and visible for a run whose job never started? Can recent runs of one workflow be listed and matched by name within the request budget? | Yes to both. **Failed** on 2026-10-18 (`trials/P-17.json`): listed at once, named within 3.7 s; decision 0036 covers the window |
| P-18 | Does a final step still run, and upload partial usage, when a job times out or is cancelled? And does a job whose step exits non-zero conclude `failure`, whether its Run refused before its work or after it? | Yes, or duration is the only estimate; a job whose step fails concludes `failure` |
| P-18 | Does a final step still run, and upload partial usage, when a job times out or is cancelled? And does a job whose step exits non-zero conclude `failure`, whether its Run refused before its work or after it? | Yes, or duration is the only estimate; a job whose step fails concludes `failure`. P-42 answered a part: a job GitHub stops at its own timeout concludes `cancelled`, not `timed_out` (`trials/P-42.json`), and a step stopped at its own `timeout-minutes` concludes `failure`, its job with it, a later `if: always()` step still running (`trials/P-42.step-timeout.json`). Whether that step uploads partial usage, and the rest of this row, are still asked |
| P-19 | Can a static page in a separate public repo be published from files the Controller writes? | Yes |
| P-20 | Does an event that anyone may cause, such as starring a repo, still start a workflow, and how often may it? | Yes, often enough for a tick every minute or two. **Passed** on 2026-10-21 (`trials/P-20.json`): ten stars a minute apart started ten runs, each within two seconds |
| P-26 | Can the model CLI's sandbox stop a tool reading the parent process's environment, and limit where it connects? | Yes to both, or the first and third layers of §4.9 carry the weight. **Failed** on 2026-10-21 (`trials/P-26.json`), at 2.1.306: in both configurations the sandbox limited connections by name, the denied host refused at CONNECT and, around its proxy, neither host's name was looked up (curl exit 6); a connection to an address with no name looked up was not tried, and the script now tries one, but the tool's shell held the parent's environment: with the scrub the listed marker was gone and the unlisted one stayed, the environment three lines shorter, the names not recorded. The sandbox would not run `ps` (exit 126), so no process was read, not even the probe's own; Linux's /proc is P-50's |
Expand Down
Loading
Loading