Repository navigation
Trial P-13 (pass): the model CLI returns validated structured output and usage, headless, at 2.1.302 #177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Trial P-13 (pass): the model CLI returns validated structured output and usage, headless, at 2.1.302 #177
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
a0fa60b
Trial P-13: the script that asks the vendor's CLI for a validated Res…
don-petry e612310
Trial P-13 (pass): the CLI returns validated structured output and us…
don-petry 74c0491
Merge origin/main into trials/p-13-cli
don-petry f2d82af
Decision 0034 becomes 0033: P-13 takes 0033, and #26 takes 0034 when …
don-petry c0df624
Merge origin/main into trials/p-13-cli
don-petry bbe7d0a
P-13, after review: a null repo goes with a subscription alone, and a…
don-petry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
51 changes: 51 additions & 0 deletions
51
...he-maintainers-own-subscription-and-a-trial-of-a-vendors-cli-records-no-repo.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # 0033. The Factory's model credential is, for now, the Maintainer's own subscription; a trial of a vendor's CLI records no repo | ||
|
|
||
| Status: accepted. Taken as a working ruling on 2026-10-18, which the Maintainer may overturn in review, for | ||
| "Trial P-13: the model CLI returns validated structured output on the subscription" (issue 15). | ||
|
|
||
| ## Decision | ||
|
|
||
| For now, the Factory's model credential is the Maintainer's own subscription. It is logged in under | ||
| a configuration directory of its own, `~/.config/sf-factory-claude`, apart from the one his | ||
| interactive sessions use, and only the Factory uses that directory. This departs from the design. | ||
| §1.1 takes the credential as "a subscription of the Factory's own", and §4.9 counts on "a | ||
| credential that belongs to the Factory alone, so that theft is a rotation and not the loss of the | ||
| Maintainer's own account". Trial P-13 ran on this login. | ||
|
|
||
| Slice 1's CI credential reopens this decision. That is the credential the `run` job holds as an | ||
| environment secret, after P-3 and P-26. Until then no workflow holds this login, and no repo | ||
| stores it. | ||
|
|
||
| The same ruling decides how a trial is recorded when it asks a vendor's CLI and not GitHub, as P-13 | ||
| does. Its fixture records `repo` as null. Each of its requests is made `as` the kind | ||
| `subscription`: the vendor account the CLI is logged in to. That account is named in words and | ||
| never by its address or a credential. `schemas/trial.schema.json` accepts null for `repo` and | ||
| `subscription` for `as.kind`, and nothing more. The schema cannot tie the two together, because | ||
| its validator, `tools/jsonschema_lite.py`, has no conditional keyword. So `tools/check_trials.py` | ||
| ties them, in a fixture and in an example alike: | ||
| - `repo` is null exactly when every request asks as `subscription`; | ||
| - a name that holds an `@` is refused as an address. | ||
|
|
||
| ## Why | ||
|
|
||
| This is the Maintainer's choice. Whether automated use of a subscription is within the vendor's | ||
| terms is not confirmed (the comments on that issue, and §6.1, last paragraph). The Maintainer accepted | ||
| that risk. It costs this: if the credential is stolen, the Maintainer loses his own account, not | ||
| only a rotated credential. That is the loss §4.9's third layer was there to prevent. Its own | ||
| configuration directory keeps the Factory's use apart from his sessions, and makes it readable as | ||
| the Factory's. It does not make the account the Factory's. | ||
|
|
||
| A fixture names where a trial ran. A trial of a vendor's CLI ran in no repo, so the canary would | ||
| be false, and null is true. None of GitHub's kinds of actor is the vendor's account, so it is given | ||
| a kind of its own. | ||
|
|
||
| ## Boundaries | ||
|
|
||
| | Boundary | Check | | ||
| |---|---| | ||
| | A trial that asks a vendor's CLI records `repo` as null, and asks as `subscription` | `examples/trial.a-vendors-cli.json` | | ||
| | `repo` is null exactly when every request asks as `subscription`: a null `repo` with any other kind is refused, and so is `subscription` with a repo | `tests/test_tools.py::test_a_null_repo_goes_with_a_subscription_and_nothing_else` | | ||
| | A fixture's `repo` is the canary or null, never another repo | `examples/invalid/trial.run-outside-the-canary.json` | | ||
| | Whoever asked is named in words, never by an address | `tests/test_tools.py::test_an_asker_named_by_an_address_is_caught` | | ||
| | A fixture of a vendor's CLI holds no credential and no header a fixture may not keep | `tools/check_trials.py` | | ||
| | The Factory's model credential is its own again, held by the `run` job as an environment secret | waiting on `harness/` | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| { | ||
| "schema": "trial/1", | ||
| "trial": "P-13", | ||
| "question": "Does the vendor CLI return validated structured output and usage in headless mode on the subscription credential, at a pinned version?", | ||
| "pass": "Yes", | ||
| "repo": null, | ||
| "ran_at": "2026-10-18T18:01:19Z", | ||
| "exchanges": [ | ||
| { | ||
| "request": { | ||
| "as": { | ||
| "kind": "subscription", | ||
| "name": "the subscription the CLI is logged in to, under a configuration directory of its own" | ||
| }, | ||
| "via": "cli", | ||
| "call": "claude -p TASK --json-schema SCHEMA --output-format json", | ||
| "headers": {}, | ||
| "body": { | ||
| "prompt": "Return the Result of run r_1 as the structured output.", | ||
| "json_schema": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "schema", | ||
| "run", | ||
| "status" | ||
| ], | ||
| "properties": { | ||
| "schema": { | ||
| "const": "result/1" | ||
| }, | ||
| "run": { | ||
| "type": "string", | ||
| "pattern": "^r_[a-z0-9]+$" | ||
| }, | ||
| "status": { | ||
| "enum": [ | ||
| "completed", | ||
| "incomplete", | ||
| "failed", | ||
| "refused", | ||
| "superseded" | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "response": { | ||
| "status": 0, | ||
| "headers": {}, | ||
| "body": { | ||
| "type": "result", | ||
| "subtype": "success", | ||
| "is_error": false, | ||
| "usage": { | ||
| "input_tokens": 2, | ||
| "output_tokens": 40 | ||
| }, | ||
| "structured_output": { | ||
| "schema": "result/1", | ||
| "run": "r_1", | ||
| "status": "completed" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ], | ||
| "removed": [ | ||
| "exchanges[0].response.body.modelUsage.some-model.inputTokens" | ||
| ], | ||
| "verdict": "pass" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.