Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .claude/skills/run-a-platform-trial/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ everything a trial leaves in it: its runs, logs and artifacts.
`GH_REPO=combsmith/software-factory-canary` for every command of the trial, and name no other
repo. P-8 needs twelve repos, P-11 the control room and the hub, P-19 a separate public repo,
P-30 the Factory's own repo, and P-31 every repo an App is installed on: for these, stop and
raise it with the Maintainer before anything runs. The fixture's schema accepts the canary
alone, so such a trial needs that decided first.
raise it with the Maintainer before anything runs. The fixture's schema accepts the canary,
or, for a vendor's CLI, no repo (decision 0033), so such a trial needs that decided first.
- **Nothing here reads the canary.** Only a document, a fixture, the fixture's example and its
schema name it here, and so does `config/apps.json`, which declares where the Apps are
installed and which nothing runs. `tools/check_trials.py` refuses any other file that names
Expand All @@ -32,6 +32,11 @@ everything a trial leaves in it: its runs, logs and artifacts.
Maintainer is advised to give a trial session a fine-grained token scoped to the canary alone,
installed outside the session. A step that must run as another identity (the machine user, an
App, a workflow's token) runs in a workflow in the canary or at the Maintainer's own terminal.
P-13 asks the vendor's CLI, not GitHub: `tools/trial_p13_cli.sh` runs it at the pinned version,
on a configuration directory of its own logged in to the Maintainer's own subscription (decision
0033), and leaves
only redacted answers, without the model's text, in a new directory outside the repo. Its
fixture records `repo` as null and asks as `subscription` (decision 0033).
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- **Real credentials wait.** Nothing stores a real credential until P-3's fixture has the verdict
`pass`, which means a secret restricted to the default branch was unreadable from a work
branch. Until then a secret's value is a dummy. No long-lived credential is put in the canary.
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/software-factory-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -1941,7 +1941,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is
| P-10 | Does a nudge count against the content-creation limit, and can its credential be dispatch-only? | Does not count; can |
| P-11 | Can the Controller's App write the control-room repo, and publish its page, without starting a workflow in the hub? | Yes |
| P-12 | Does a check run's `external_id` round-trip, so it can serve as a recovery key? | Yes |
| P-13 | Does the vendor CLI return validated structured output and usage in headless mode on the subscription credential, at a pinned version? | Yes |
| P-13 | Does the vendor CLI return validated structured output and usage in headless mode on the subscription credential, at a pinned version? Run by `tools/trial_p13_cli.sh` at the Maintainer's terminal, on a configuration directory of its own logged in to the Maintainer's own subscription (decision 0033), with one fixed Task and the smallest Result of §3.5 as its schema | Yes: the structured output validates, the same answer carries input and output tokens, and it ran at the pinned version. Until the engine port in `adapters/` exists, the trial script pins the version. If not, the metered key is tried and recorded. **Passed** on 2026-10-18 against the script's pin, 2.1.302 (`trials/P-13.json`), on the Maintainer's own subscription under a configuration directory of its own, which departs from §1.1 and §4.9 for now (decision 0033) |
| P-14 | Can a job with no permissions and no cache action write the default-branch cache scope? | Cannot |
| P-15 | Do the retained Advisory reviewers review a pull request authored by an App? | Each one does, or is not counted as a Lens on such pull requests |
| P-16 | How long are run records, logs and artifacts kept in a public repo? | 90 days |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# 0033. The Factory's model credential is, for now, the Maintainer's own subscription; a trial of a vendor's CLI records no repo

Status: accepted. Taken as a working ruling on 2026-10-18, which the Maintainer may overturn in review, for
"Trial P-13: the model CLI returns validated structured output on the subscription" (issue 15).

## Decision

For now, the Factory's model credential is the Maintainer's own subscription. It is logged in under
a configuration directory of its own, `~/.config/sf-factory-claude`, apart from the one his
interactive sessions use, and only the Factory uses that directory. This departs from the design.
§1.1 takes the credential as "a subscription of the Factory's own", and §4.9 counts on "a
credential that belongs to the Factory alone, so that theft is a rotation and not the loss of the
Maintainer's own account". Trial P-13 ran on this login.

Slice 1's CI credential reopens this decision. That is the credential the `run` job holds as an
environment secret, after P-3 and P-26. Until then no workflow holds this login, and no repo
stores it.

The same ruling decides how a trial is recorded when it asks a vendor's CLI and not GitHub, as P-13
does. Its fixture records `repo` as null. Each of its requests is made `as` the kind
`subscription`: the vendor account the CLI is logged in to. That account is named in words and
never by its address or a credential. `schemas/trial.schema.json` accepts null for `repo` and
`subscription` for `as.kind`, and nothing more. The schema cannot tie the two together, because
its validator, `tools/jsonschema_lite.py`, has no conditional keyword. So `tools/check_trials.py`
ties them, in a fixture and in an example alike:
- `repo` is null exactly when every request asks as `subscription`;
- a name that holds an `@` is refused as an address.

## Why

This is the Maintainer's choice. Whether automated use of a subscription is within the vendor's
terms is not confirmed (the comments on that issue, and §6.1, last paragraph). The Maintainer accepted
that risk. It costs this: if the credential is stolen, the Maintainer loses his own account, not
only a rotated credential. That is the loss §4.9's third layer was there to prevent. Its own
configuration directory keeps the Factory's use apart from his sessions, and makes it readable as
the Factory's. It does not make the account the Factory's.

A fixture names where a trial ran. A trial of a vendor's CLI ran in no repo, so the canary would
be false, and null is true. None of GitHub's kinds of actor is the vendor's account, so it is given
a kind of its own.

## Boundaries

| Boundary | Check |
|---|---|
| A trial that asks a vendor's CLI records `repo` as null, and asks as `subscription` | `examples/trial.a-vendors-cli.json` |
| `repo` is null exactly when every request asks as `subscription`: a null `repo` with any other kind is refused, and so is `subscription` with a repo | `tests/test_tools.py::test_a_null_repo_goes_with_a_subscription_and_nothing_else` |
| A fixture's `repo` is the canary or null, never another repo | `examples/invalid/trial.run-outside-the-canary.json` |
| Whoever asked is named in words, never by an address | `tests/test_tools.py::test_an_asker_named_by_an_address_is_caught` |
| A fixture of a vendor's CLI holds no credential and no header a fixture may not keep | `tools/check_trials.py` |
| The Factory's model credential is its own again, held by the `run` job as an environment secret | waiting on `harness/` |
73 changes: 73 additions & 0 deletions examples/trial.a-vendors-cli.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
{
"schema": "trial/1",
"trial": "P-13",
"question": "Does the vendor CLI return validated structured output and usage in headless mode on the subscription credential, at a pinned version?",
"pass": "Yes",
"repo": null,
"ran_at": "2026-10-18T18:01:19Z",
"exchanges": [
{
"request": {
"as": {
"kind": "subscription",
"name": "the subscription the CLI is logged in to, under a configuration directory of its own"
},
"via": "cli",
"call": "claude -p TASK --json-schema SCHEMA --output-format json",
"headers": {},
"body": {
"prompt": "Return the Result of run r_1 as the structured output.",
"json_schema": {
"type": "object",
"additionalProperties": false,
"required": [
"schema",
"run",
"status"
],
"properties": {
"schema": {
"const": "result/1"
},
"run": {
"type": "string",
"pattern": "^r_[a-z0-9]+$"
},
"status": {
"enum": [
"completed",
"incomplete",
"failed",
"refused",
"superseded"
]
}
}
}
}
},
"response": {
"status": 0,
"headers": {},
"body": {
"type": "result",
"subtype": "success",
"is_error": false,
"usage": {
"input_tokens": 2,
"output_tokens": 40
},
"structured_output": {
"schema": "result/1",
"run": "r_1",
"status": "completed"
}
}
}
}
],
"removed": [
"exchanges[0].response.body.modelUsage.some-model.inputTokens"
],
"verdict": "pass"
}
12 changes: 8 additions & 4 deletions schemas/trial.schema.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "The fixture of one platform trial: what was asked of GitHub in the canary repo, what it answered, and the verdict.",
"title": "The fixture of one platform trial: what was asked of GitHub in the canary repo, or of a vendor's CLI in no repo (decision 0033), what it answered, and the verdict.",
"$comment": "A credential is refused by tools/check_trials.py, not here: this validator has no not and no propertyNames, and its pattern refuses every string that ends in a newline, so a pattern that refused token shapes would refuse a recorded log line too.",
"type": "object",
"additionalProperties": false,
Expand Down Expand Up @@ -35,8 +35,11 @@
"minLength": 1
},
"repo": {
"description": "Every trial runs in the canary repo.",
"const": "combsmith/software-factory-canary"
"description": "Where the trial ran: the canary repo, or null for a trial that asks a vendor's CLI and not GitHub, such as P-13 (decision 0033).",
"enum": [
"combsmith/software-factory-canary",
null
]
},
"ran_at": {
"$ref": "#/$defs/time"
Expand Down Expand Up @@ -104,7 +107,8 @@
"machine_user",
"app",
"workflow",
"anonymous"
"anonymous",
"subscription"
]
},
"name": {
Expand Down
33 changes: 33 additions & 0 deletions tests/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -1978,6 +1978,39 @@ def test_every_example_of_a_fixture_and_the_folder_notes_are_scanned(self):
self.assertTrue(caught(problems, "examples/trial.json: note holds something shaped like a credential"))
self.assertTrue(caught(problems, "trials/AGENTS.md: holds something shaped like a credential"))

def test_a_null_repo_goes_with_a_subscription_and_nothing_else(self):
# Decision 0033. The schema cannot tie the two together, so the check does, both ways.
vendor = json.loads((ROOT / "examples" / "trial.a-vendors-cli.json").read_text())
self.assertEqual(check_trials.run(clone({"trials/P-13.json": json.dumps(vendor)})), [])
problems = check_trials.run(clone({"trials/P-12.json": fixture(repo=None)}))
self.assertTrue(caught(problems, "trials/P-12.json: exchanges[0].request.as.kind is app, and repo is null"))
self.assertTrue(caught(problems, "trials/P-12.json: exchanges[1].request.as.kind is app, and repo is null"))
inside = dict(vendor, repo="combsmith/" + check_trials.CANARY)
problems = check_trials.run(clone({"trials/P-13.json": json.dumps(inside)}))
self.assertTrue(caught(problems, "trials/P-13.json: exchanges[0].request.as.kind is subscription, and repo is"))
mixed = json.loads(json.dumps(vendor))
mixed["exchanges"].append(json.loads(json.dumps(mixed["exchanges"][0])))
mixed["exchanges"][1]["request"]["as"] = {"kind": "person", "name": "don-petry"}
problems = check_trials.run(clone({"trials/P-13.json": json.dumps(mixed)}))
self.assertTrue(caught(problems, "exchanges[1].request.as.kind is person, and repo is null"))
self.assertFalse(caught(problems, "exchanges[0].request.as.kind"))
problems = check_trials.run(clone({"examples/trial.json": fixture(repo=None)}))
self.assertTrue(caught(problems, "examples/trial.json: exchanges[0].request.as.kind is app, and repo is null"))

def test_an_asker_named_by_an_address_is_caught(self):
# Decision 0033: a fixture names who asked in words, never by an address that ties it to an account.
vendor = json.loads((ROOT / "examples" / "trial.a-vendors-cli.json").read_text())
vendor["exchanges"][0]["request"]["as"]["name"] = "account@example.com"
problems = check_trials.run(clone({"trials/P-13.json": json.dumps(vendor),
"examples/trial.a-vendors-cli.json": json.dumps(vendor)}))
for where in ("trials/P-13.json", "examples/trial.a-vendors-cli.json"):
self.assertTrue(caught(problems, where + ": exchanges[0].request.as.name looks like an address"), where)
data = json.loads(fixture())
data["exchanges"][1]["request"]["as"]["name"] = "combsmith-sf-trial@users.noreply.github.com"
problems = check_trials.run(clone({"trials/P-12.json": json.dumps(data)}))
self.assertTrue(caught(problems, "trials/P-12.json: exchanges[1].request.as.name looks like an address"))
self.assertFalse(caught(problems, "exchanges[0].request.as.name"))

def test_code_that_reads_the_canary_is_caught(self):
name = "combsmith/" + check_trials.CANARY
root = clone({"tools/fetch.py": "REPO = %r\n" % name, ".github/workflows/trial.yml": "repository: %s\n" % name,
Expand Down
43 changes: 41 additions & 2 deletions tools/check_trials.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,39 @@ def _credentials(value, where: str) -> list:
return out


def _askers(data) -> list:
"""(path, what is wrong) for who asked, beyond what the schema can say (decision 0033).

A trial of a vendor's CLI ran in no repo: `repo` is null exactly when every request is made as
`subscription`. And an asker is named in words, never by an address, which could tie the fixture
to an account at a vendor.
"""
out = []
exchanges = data.get("exchanges") if isinstance(data, dict) else None
exchanges = exchanges if isinstance(exchanges, list) else []
kinds = []
for i, exchange in enumerate(exchanges):
request = exchange.get("request") if isinstance(exchange, dict) else None
asker = request.get("as") if isinstance(request, dict) else None
asker = asker if isinstance(asker, dict) else {}
kinds.append(asker.get("kind"))
if "@" in str(asker.get("name", "")):
out.append(("exchanges[%d].request.as.name" % i, "looks like an address; name the account in words, "
"never by its address"))
if isinstance(data, dict) and "repo" in data:
if data["repo"] is None:
for i, kind in enumerate(kinds):
if kind != "subscription":
out.append(("exchanges[%d].request.as.kind" % i, "is %s, and repo is null; only a trial of a "
"vendor's CLI, every request of it made as subscription, records no repo" % kind))
else:
for i, kind in enumerate(kinds):
if kind == "subscription":
out.append(("exchanges[%d].request.as.kind" % i, "is subscription, and repo is %s; a trial of a "
"vendor's CLI ran in no repo, so its repo is null" % data["repo"]))
return out


def _headers(data) -> list:
"""(path, what is wrong) for each header of an exchange that a fixture may not keep."""
out = []
Expand Down Expand Up @@ -175,6 +208,8 @@ def _fixtures(root: pathlib.Path, schema: dict, problems: list) -> dict:
wrong = ["%s: %s" % (rel, err) for err in validate(data, schema)]
if not wrong and data["trial"] != name.group(1):
wrong.append("%s: records trial %s; a fixture is named for its trial" % (rel, data["trial"]))
if not wrong:
wrong += ["%s: %s %s" % (rel, where, what) for where, what in _askers(data)]
wrong += ["%s: %s %s" % (rel, where, what) for where, what in _credentials(data, "") + _headers(data)]
problems += wrong
if not wrong:
Expand All @@ -183,7 +218,8 @@ def _fixtures(root: pathlib.Path, schema: dict, problems: list) -> dict:


def _examples(root: pathlib.Path) -> list:
"""The examples of a fixture, counter-examples too, hold no credential either."""
"""The examples of a fixture, counter-examples too, hold no credential either; an example names its askers as a
fixture must."""
problems = []
for path in walk.files(root, "examples"):
if not (path.name.startswith("trial") and path.suffix == ".json"):
Expand All @@ -193,7 +229,10 @@ def _examples(root: pathlib.Path) -> list:
data = _load(path)
except (OSError, ValueError):
continue # check_examples reports it
problems += ["%s: %s %s" % (rel, where, what) for where, what in _credentials(data, "") + _headers(data)]
found = _credentials(data, "") + _headers(data)
if "invalid" not in path.relative_to(root).parts:
found += _askers(data)
problems += ["%s: %s %s" % (rel, where, what) for where, what in found]
return problems


Expand Down
6 changes: 6 additions & 0 deletions tools/mutants.py
Original file line number Diff line number Diff line change
Expand Up @@ -953,6 +953,12 @@ def moved_after(rel: str, block: str, anchor: str) -> tuple:
'NEVER = ("workflows", "security_events")', 'NEVER = ("security_events",)'),
("an App may hold security_events", "tools/check_apps.py",
'NEVER = ("workflows", "security_events")', 'NEVER = ("workflows",)'),
("a fixture may name who asked by an address", "tools/check_trials.py",
' if "@" in str(asker.get("name", "")):\n', " if False:\n"),
("a fixture with no repo may ask as anyone", "tools/check_trials.py",
' if kind != "subscription":\n', " if False:\n"),
("a subscription may ask in a fixture of a repo", "tools/check_trials.py",
' if kind == "subscription":\n', " if False:\n"),
("the impostor workflow posts from the Lens's own file", "sim/world.py",
'"workflow": IMPOSTOR_WORKFLOW,', '"workflow": WORKFLOWS[lens],'),
("a neutral check is read as passing", "config/defaults.json",
Expand Down
Loading
Loading