Skip to content

Trials P-59 and P-60: the read tools held to their folder, and a tarball's one redirect to codeload - #532

Merged
don-petry merged 4 commits into
mainfrom
claude/p59-read-confinement
Nov 4, 2026
Merged

don-petry merged 4 commits into
mainfrom
claude/p59-read-confinement

Conversation

@don-petry

@don-petry don-petry commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Refs #537. Trials P-59 and P-60, both pass. They are the evidence for the design in which the review Role reads a pull request's head. That design's record, 0138, is not in this pull request.

P-60: a commit's tarball is one redirect to codeload

The Maintainer approved "Tarball, allow codeload" on 2026-11-04. P-60 records what that redirect is, with every request made without a credential against the public canary:

Ask Answer
GET /repos/{canary}/tarball/{sha}, following no redirect 302 to https://codeload.github.com/{owner}/{repo}/legacy.tar.gz/{sha}, with no token in the Location
That Location, sent with no Authorization header 200, application/x-gzip, 4,337 bytes, with no content-length. The tar opens with a pax global header whose comment is the whole commit sha, then one folder holding only directories and regular files
Codeload over plain HTTP 301 to the same address over HTTPS, the only redirect codeload gave
Codeload with the owner and repo in capitals, or with the branch name for the sha 200, the same archive
A commit the canary does not hold The API still answers 302; codeload answers 404

What 0138 takes from this:

  • The transport may follow one redirect to codeload and must refuse a second hop.
  • It holds its own size bound, since codeload sends no content-length.
  • A head that isn't there shows up as codeload's 404.
  • The unpacker can check the pax comment against the commit it asked for.

The fixture keeps the archive's bytes in base64, so 0138's tests can unpack exactly what GitHub served.

Owed: what codeload's Location holds for a private repo read with an App's token. A live Run on a private head waits on it; the eval doesn't, because its cases' repo is public.

P-59: the read tools alone, held to their folder

The question. On a GitHub Linux runner, with bubblewrap and the subprocess scrub set as the run job sets them, is the pinned CLI (2.1.306), given only the read tools, held to its working folder?

  • Is every read outside that folder refused?
  • Do the folder's own .claude settings, CLAUDE.md or AGENTS.md take effect?
  • Does any write land?

How it ran. In the canary, run 41772353972, after the reads' fix round, with no secret and no model: a stand-in model, P-50's stub, ran on the runner itself. The CLI was installed by its checksum, as the run job installs it, and no sysctl was changed. The run before the fix round, 41756788469, also passed; it is kept in the fixture and superseded.

The configuration a review Run would be given:

--tools Read,Grep,Glob --allowedTools Read,Grep,Glob --permission-mode dontAsk --restricted --setting-sources ""
Probe engine (both flags) --restricted alone --setting-sources "" alone neither
Read inside the folder (control) ok ok ok ok
18 calls outside: absolute, .., three links, /proc/self/cwd through links, /proc/self/root, the home directory, by Read, Grep and Glob all refused all refused read read
/proc/self/environ and the parent's environ, by Read refused refused refused refused
The same, by Grep refused refused read the CLI's environment read the CLI's environment
Bash, Write not tools not tools not tools not tools
The head's two hooks and its apiKeyHelper did not run did not run did not run all three ran
The head's CLAUDE.md and AGENTS.md sent to the model no no no yes
Verdict pass would pass fails fails

The verdict is decided by the engine configuration alone; the other three are recorded.

What the four configurations show:

  • --restricted is what confines the reads.
  • --setting-sources "" alone holds no read. With the scrub set, the CLI forces the permission mode back to its default, and the allowed tools, named without a path, then read anywhere.
  • Either flag keeps the head's settings and instructions out.
  • A real Run's environment holds its credential, so a review Run is given both flags.

The trial's other findings:

  • With the scrub set, the CLI forces the permission mode back to its default unless the allowed tools are named, so they are named.
  • Grep does not follow a link inside the folder.
  • A smoke run on a Mac gave the same results, apart from the /proc reads, since a Mac has no /proc.

What this pull request changes. Every row below that mentions P-59 is new.

Part Change
tools/trial_p59_read_confinement.sh The trial's runner. Its pass condition was written before the run, at the top of the file. It runs only in the canary's workflow
tools/trial_p50_stub.py When calls.json stands where the stub runs, it answers those calls in turn, and logs the tools it offered and the markers each result held, by name only. Without that file it behaves exactly as P-50's stub did, and its check_repo allowlist is unchanged
trials/P-59.json The fixture: three pushes and their runs, the first refused by the script's own guard since OUT was under /home, the second superseded by the fix round, and the third the trial's run; that run's job, the three files as they ran, the log, and the analysis. Each file the canary ran has the same git blob as the file in tools/
tests/test_tools.py ReadConfinementTrial: a stand-in CLI that holds the reads to the folder under --restricted, and keeps the settings out under either flag, as the pinned CLI did, plus flags that make each part of the condition fail on its own. One test covers the stub's scripted mode, and one re-reads the fixture's analysis from its log
trials/P-60.json P-60's seven exchanges, the archive among them
Section 6 and trials/AGENTS.md Section 6 gains P-59's and P-60's rows, and trials/AGENTS.md P-59's rules with the test that holds each. A test re-reads P-60: the one redirect, codeload's answer with no Authorization header, the tar's pax comment and its members, and no redirect from codeload over HTTPS
tools/mutants.py P-59's 18 mutants: one for each part of the condition, one for each flag, and two for the stub

Grounding. No rule in sim/grounding.py asks this trial, so nothing is grounded.

Canary, undone. The branch trial/p-59 is deleted. The three runs stay, as P-50's and P-51's did; their logs hold no secret, and the redactor removed one line, line 93.

Removed from the fixture: exchanges[1,3,5,6,7,8,9].request.headers.authorization, and line 93 of the log in exchanges[10].response.body.

Verify

  • make verify passed locally at 710400a9. At 5dd3c9ea, the fix round's head, it passed too.
  • At 5dd3c9ea, make checks passed, and so did ReadConfinementTrial and both fixture tests.
  • python3 -m tools.mutants tools/trial_p59_read_confinement.sh tools/trial_p50_stub.py killed 19 of 19.
  • The whole run is CI's checks and its 8 mutant shards.

Reads

Argon, Muse Wonder Full, the Claude review and the security pass all read 490a36d. Every finding was fixed in 5dd3c9ea, in part where its disposition says so, apart from one not changed. The fix round was not read again.

Reviewer Head Unread parts Blocking Major Minor Info
agy 490a36d none 0 6 7 0
muse 490a36d none 0 2 1 0
claude-review 490a36d none 0 1 3 0
claude-security 490a36d none 0 0 3 0
Finding Severity Criterion Where What Disposition
agy#1 major PR-3 tests/test_tools.py:5272 P-59 test skips calls[0] on outside-read check and passes on vacuous call lists Fixed in 5dd3c9e: the fixture test counts the 18 outside calls and holds each refused, and each part's 24 calls and 3 write calls by is True
agy#2 major PR-3 tests/test_tools.py:5314 P-60 HTTPS redirect loop can pass vacuously and leaves ex[3..4] and probe URLs unchecked Fixed in 5dd3c9e: each codeload address parsed with urllib.parse, exchanges 3 and 4 checked, any 3xx counted
agy#3 major PR-3 tests/test_tools.py:7973 ReadConfinementTrial has no case where reading inside the working folder fails Fixed in 5dd3c9e, as the Claude review's #1: a refused inside read, with its mutant
agy#4 major PR-3 tests/test_tools.py:8041 List-shaped tool_result content in the stub test carries no marker, leaving list marker extraction unheld Fixed in 5dd3c9e: the stub test's list-shaped result carries a marker
agy#5 major PR-3 tests/test_tools.py:8065 Pre-run CLI/version refusals are untested in test_each_refusal_stops_before_anything_is_asked, and refusal runs inherit host env Fixed in part in 5dd3c9e: refusals run in an environment of their own, and a missing CLI is refused. The script refuses no version before the call; it records it and fails the verdict, as the moved-version case holds
agy#6 major none trials/P-59.json:5 Trial P-59 leaves Grep/Glob symlink paths, parent environ Grep, and Glob filename leaks untested Fixed in 5dd3c9e: Grep and Glob through the links and of the parent's environ, an outside file named for its marker; P-59 run again
agy#7 minor PR-3 tests/test_tools.py:5264 P-59 copied_whole check passes on a single-key dict without a tools/ entry Fixed in 5dd3c9e: both keys checked, and each blob compared with git's blob of the file on disk
agy#8 minor PR-3 tests/test_tools.py:5267 P-59 test is named only_with_both_flags but checks only 'engine' and 'unguarded' Fixed in 5dd3c9e: the restricted and sources configurations, each held by the fixture test
agy#9 minor PR-3 tests/test_tools.py:5293 P-60 checks 'authorization' and 'content-length' headers case-sensitively Fixed in 5dd3c9e: every header name in the fixture is held to lower case
agy#10 minor PR-3 tests/test_tools.py:7966 ReadConfinementTrial never asserts cleanup of TMPDIR Fixed in 5dd3c9e: a TMPDIR of each run's own, held empty after
agy#11 minor PR-3 tests/test_tools.py:7983 write-lands fails no_write_tool and nothing_written only together Fixed in 5dd3c9e: a write that lands but says it failed fails nothing_written alone
agy#12 minor none trials/P-59.json:5 Removing both flags together in "unguarded" does not show that each flag individually matters Fixed in 5dd3c9e: each flag run alone; --setting-sources "" alone held no read
agy#13 minor none trials/P-60.json:46 Anonymous codeload exchanges record request.as as person don-petry instead of anonymous Not changed: tools/check_trials.py requires a command-line read to be the Maintainer's own; each codeload call says it sent no credential
muse#1 major none tools/trial_p59_read_confinement.sh:240 P-59 passes when Glob outside the folder succeeds, since the judge only looks for marker text Fixed in 5dd3c9e, as the security pass's #1
muse#2 major PR-3 tools/trial_p59_read_confinement.sh:245 Mutant setting P-59's inside_read to True survives: no case denies the inside read Fixed in 5dd3c9e, as the Claude review's #1
muse#3 minor none tools/trial_p59_read_confinement.sh:236 P-59 judges offered tools from the first request alone, so a tool added later is unrecorded Fixed in 5dd3c9e, as the Claude review's #2
claude-review#1 major PR-3 tools/trial_p59_read_confinement.sh:245 Three parts of P-59's pass condition are held by no test; their mutants survive ReadConfinementTrial Fixed in 5dd3c9e: a refused inside read, a Task ended early and Bash answered, each a case and a mutant
claude-review#2 minor PR-1 tools/trial_p50_stub.py:9 The PR closes no issue, and the issue the stub names, 517, is unrelated Fixed in 5dd3c9e: every request's tools, with a late-tool case and a mutant
claude-review#3 minor PR-3 tools/trial_p59_read_confinement.sh:186 The /proc/self/cwd probe normalises to a path that does not exist, so it cannot show confinement Fixed in 5dd3c9e: the /proc/self/cwd probes go through links; run again, refused under --restricted and read without it
claude-review#4 minor PR-3 tools/trial_p59_read_confinement.sh:236 read_tools_alone reads only the first request's tools, so a tool offered later in the conversation passes Fixed in part in 5dd3c9e: the stub names the Maintainer's decision of 2026-11-04. This PR refs #537, opened for the work; 0138's PR closes it
claude-security#1 minor PR-3 tools/trial_p59_read_confinement.sh:240 P-59's verdict cannot see a Glob that lists the outside folder: a Glob escape still passes Fixed in 5dd3c9e: each outside call must be an error, and an outside file is named for its marker
claude-security#2 minor PR-3 tools/trial_p59_read_confinement.sh:240 An outside marker that reaches the model outside a tool result is recorded but never fails the engine Fixed in 5dd3c9e: a marker sent anywhere to the model counts
claude-security#3 minor PR-3 tools/trial_p59_read_confinement.sh:243 Two guards against a vacuous P-59 pass are held by no test: every_call_answered and inside_read mutants survive Fixed in 5dd3c9e, as the Claude review's #1

Bots:

  • CodeQL, "Incomplete URL substring sanitization" in the P-60 test: fixed in 5dd3c9ea. Each address is parsed with urllib.parse and its host compared whole.
  • SonarCloud S5131, "Reflected XSS via file content", at tools/trial_p50_stub.py:161: not a vulnerability. The stub answers JSON, Content-Type: application/json, on 127.0.0.1, to the CLI under trial. No browser reads it, and the content is the trial's own calls, which the script writes. The gate is not required. Marking it safe in Sonar is the Maintainer's to do.
  • Codex hit its quota, and CodeRabbit skipped the PR: auto reviews are off.

🤖 Generated with Claude Code

don-petry and others added 2 commits November 3, 2026 19:58
…en both flags

On a GitHub Linux runner with bubblewrap and the scrub, as the run job has
them, the pinned CLI given --tools Read,Grep,Glob --allowedTools
Read,Grep,Glob --permission-mode dontAsk --restricted --setting-sources ""
refused every read outside its folder: absolute, .., three links,
/proc/self/environ, its parent's environ, /proc/self/cwd, /proc/self/root
and the home directory. The folder's own .claude settings did nothing and
its CLAUDE.md was not sent. Without the two flags, the same settings ran two
hooks and a key helper, opened the outside reads, and Grep read the CLI's own
environment.

tools/trial_p59_read_confinement.sh runs it in the canary alone, against
P-50's stub, which now answers a list of calls from calls.json when one is
there and logs markers by name alone; with none it is P-50's stub as it was.
The fixture is trials/P-59.json, and the design's section 6 has its row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…59's mutants

Asked with no credential, the API answered a tarball of a canary commit 302
to codeload.github.com, under the repo's path and the commit, with no token
in the Location; codeload answered it, with no Authorization header, 200, a
gzip with no content-length whose tar opens with a pax global header naming
the commit. No codeload address over HTTPS redirected; plain HTTP answered
301 to HTTPS. For a commit it does not hold the API still answers 302 and
codeload 404. The fixture keeps the archive, for 0138's tests to unpack.

P-59's mutants: one for each part of its condition, each flag, and two for
the stub's script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner November 4, 2026 03:01
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-04T03:01:15.510172Z
We couldn’t complete this review. Comment @codex review to try again.
490a36d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

Comment thread tests/test_tools.py Fixed
The reads' fix round, and the trial run again in the canary:
- every call that reaches outside the folder must be an error, an outside
  file is named for its marker so a listing shows it, and a marker that
  reaches the model anywhere, not only in a tool result, counts;
- every request's tools are judged, not the first one's alone;
- the /proc/self/cwd probes go through links, which a path's normalising
  cannot turn into a file that is not there; Grep and Glob are asked
  through the links and of the parent's environ too;
- each flag is also run alone. --restricted alone held every read;
  --setting-sources "" alone kept the settings out and held no read, since
  with the scrub set the allowed tools, named without a path, read
  anywhere, and Grep read the CLI's own environment;
- the stand-in CLI covers a refused inside read, a Task ended early, Bash
  answered, a write that lands quietly, a tool offered late and a marker
  sent beside the task, each with a mutant; refusals run in an environment
  of their own and nothing is left in TMPDIR;
- the P-60 test reads each codeload address parsed, the capitals and the
  branch exchanges, and holds headers to lower case;
- the stub's docstring names the Maintainer's decision, not issue 517.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Maintainer's ruling of 2026-11-04: pythonsecurity:S5131 on
tools/trial_p50_stub.py is a false positive, waived in code rather than in
SonarCloud. The stub answers the pinned CLI on 127.0.0.1 alone, with a
json.dumps body sent as application/json. It is set aside for that one rule
and that one file, as python:S5332 already is. The stub is not edited:
P-59's fixture holds its blob.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 2689570 Nov 4, 2026
19 checks passed
@don-petry
don-petry deleted the claude/p59-read-confinement branch November 4, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants