diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 8bf66eba4..7debd2c87 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -52,3 +52,11 @@ jobs: RELEASE_TO: ${{ inputs.to }} RELEASE_CUT_COMMIT: ${{ github.sha }} RELEASE_TOKEN: ${{ steps.mint.outputs.token }} + RELEASE_EXCHANGES: ${{ github.workspace }} + - if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: release-exchanges + path: release-exchanges.json + if-no-files-found: ignore + retention-days: 90 diff --git a/docs/decisions/0176-the-release-step-keeps-its-own-exchanges-held-to-lists-as-a-public-artifact-of-its-run.md b/docs/decisions/0176-the-release-step-keeps-its-own-exchanges-held-to-lists-as-a-public-artifact-of-its-run.md new file mode 100644 index 000000000..a596005c6 --- /dev/null +++ b/docs/decisions/0176-the-release-step-keeps-its-own-exchanges-held-to-lists-as-a-public-artifact-of-its-run.md @@ -0,0 +1,36 @@ +# 0176. The release step keeps its own exchanges, held to lists, as a public artifact of its run + +Status: accepted. The Factory Builder's rulings of 2026-11-09 on the design for "The release step records its own exchanges, redacted, after factory/v0.1.0" (issue 531), which ruling 9′ of "Release: cut an immutable tag, move next, and roll back in one action" (issue 197), as amended on 2026-11-03, left open: a fixture of a Release's runs could show each run's outcome and what was read from outside it, but not what the step itself read and wrote. The edit to `.github/workflows/release.yml` was pre-approved by the Maintainer on 2026-11-08 and ruled by the Factory Builder. + +## Decision + +| What | Decided | Because | +|---|---|---| +| What is kept | Each exchange the step sends through `RELEASE_TOKEN`, by `release/exchanges.py`'s `Recording`, which wraps the step's one transport and passes each request through unchanged | Every read and write of the step already goes through that one object, so nothing it sends is missed and nothing it does changes | +| Held to lists | The call, its Accept where one is given, the headers `tools/check_trials.py`'s `HEADERS` lets a trial's fixture keep but `location`, and of a JSON body only the fields `FIELDS` names, at any depth, of what is sent as of what is answered; everything else is left out and its place listed in `dropped` | An artifact of a public repo can be downloaded by anyone signed in, so the file is public: what is kept is named, and nothing is kept for not being refused | +| Kept by its measure alone | An artifact's download, a file read at a commit, and any body that is not JSON: their size and sha256, never their text; a file at a commit by its own bytes, decoded from the base64 GitHub answers it in, so it can be held to the file; an answer with no body, by nothing | The download's redirect names a signed address; the eval report and the set's manifest are where a held-out case could appear; a gateway's page can quote anything | +| Then redacted | `tools/redact_trial.py`'s `redact_json`, the trials' own, over the whole file, each place it took something from listed in `removed` | One set of patterns for every capture the Factory keeps, which cannot drift; importing it starts nothing and reads nothing but modules | +| Never in the way | A fault in keeping drops that exchange and is counted in `unkept`; it never changes what the step is answered, does or prints. The file is written whatever stopped the step, in a `finally` | The step's codes and its writes are the release's evidence; a record of them must not become a way for them to fail | +| Where it goes | `RELEASE_EXCHANGES: ${{ github.workspace }}` names the folder, the checkout's root, where no commit may hold that file (`tools/check_repo.py`), so none stands there first to be uploaded in its place; the step writes the file `release/exchanges.py`'s `NAME` names there, never over one that stands; the job's last step, `if: always()`, uploads it as the one artifact `release-exchanges`, `if-no-files-found: ignore`, for 90 days; no step prints it | An expression stands only as a whole value in a workflow; 90 days is how long the eval job's `eval-report` and `eval-ledger` are kept (decision 0164), and a rollback, such as that issue's live one with v0.1.1, may come more than 30 days after a cut | +| Its form | `release-exchanges/1` (`schemas/release-exchanges.schema.json`), a record: its exchanges in the shape a `release-fixture/1` item keeps them, so a builder adds them to a fixture as they are, after redacting them again outside the repo | The fixture is where an item is held to its own exchanges (`tools/check_release.py`); the artifact is what it is made from | + +## Why + +Ruling 9′ of that issue keeps a fixture of each run of `.github/workflows/release.yml`, read from outside it: the run, its job's log at the step, the refs after. The step prints codes alone, so what the eval gate read, what `verify` and the comparison with `main` answered, and each write and its read-back, were seen by nothing but the step. Keeping them inside the run is the only way to see them, and the run's artifacts are public. So the design starts from what may be shown, not from what must be hidden: a field, a header or a body is kept only when a list names it, and a file's text, a signed address and an error's stray fields cannot be kept by any route but a list that does not name them. + +## Boundaries + +| Boundary | Check | +|---|---| +| A field no list names, a header no fixture keeps, a redirect's address, a download's or a file's text, and a body that is not JSON are left out of what the step keeps | `tests/test_release_exchanges.py::test_the_leak_surface_is_left_out`, `release/exchanges.py`; mutant: "the release step keeps a field no list names"; mutant: "the release step keeps a redirect's address"; mutant: "the release step keeps a header no fixture keeps"; mutant: "the release step keeps an artifact's download whole"; mutant: "the release step keeps a file at a commit whole"; mutant: "the release step keeps a body that is not JSON" | +| Every counter-example the trials' redaction is tested with is taken out of what the step keeps | `tests/test_release_exchanges.py::test_every_counter_example_of_the_trials_redaction_is_left_out_here`, `release/exchanges.py`; mutant: "the release step's exchanges are not redacted" | +| What `release/` imports of `tools/` starts nothing and reads nothing but modules at import | `tests/test_release_exchanges.py::test_importing_the_redaction_starts_nothing_and_reads_nothing_but_modules`, `release/exchanges.py`; mutant: "the release step's keeping starts a process at import" | +| A fault in keeping never changes what the step is answered | `tests/test_release_exchanges.py::test_a_fault_in_keeping_never_changes_what_the_step_is_answered`, `release/exchanges.py`; mutant: "a fault in keeping stops the release step" | +| It writes one file, in the folder it is given, never over one that stands | `tests/test_release_exchanges.py::test_it_writes_only_the_one_file_in_the_folder_the_workflow_names`, `release/exchanges.py`; mutant: "the release step writes over a file that stands" | +| Where a folder is named, the step keeps each exchange and prints what it printed before; where it stops or fails, it keeps what it sent | `tests/test_release_exchanges.py::test_a_cut_keeps_each_exchange_and_prints_what_it_printed_before`, `tests/test_release_exchanges.py::test_a_step_that_stops_or_fails_still_keeps_what_it_sent`, `release/move.py`; mutant: "the release step keeps nothing where a folder is named"; mutant: "a release step that fails keeps nothing" | +| `.github/workflows/release.yml`'s last step uploads the file whatever stopped the step, as the one artifact `release-exchanges`, kept 90 days, and nothing prints it | `tests/test_release_workflow.py::test_each_counter_example_is_refused_for_its_one_fault`, `tools/check_workflows.py`; mutant: "the release job uploads its exchanges only when the step succeeds"; mutant: "the release job keeps its exchanges 30 days" | +| What is sent is held to the lists as what is answered is, each field left out is named, and an answer with no body is measured by nothing | `tests/test_release_exchanges.py::test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named`, `release/exchanges.py`; mutant: "the release step keeps what it sends whole"; mutant: "the release step keeps every header it sends"; mutant: "a field the release step leaves out is not named"; mutant: "an answer with no body is measured as null" | +| A file at a commit is measured as its own bytes | `tests/test_release_exchanges.py::test_a_file_at_a_commit_is_measured_as_the_file_itself`, `release/exchanges.py`; mutant: "a file at a commit is measured as GitHub's envelope" | +| No commit holds the file the step writes at the checkout's root | `tests/test_tools.py::test_no_commit_holds_the_file_the_release_step_writes_at_the_root`, `tools/check_repo.py`; mutant: "a committed release-exchanges.json at the root passes" | +| A text, a number or a truth is kept only under a field a list names, never alone or in a list no field names; a left-out field's name is written only where it is a plain identifier | `tests/test_release_exchanges.py::test_a_value_is_kept_only_under_a_field_a_list_names`, `release/exchanges.py`; mutant: "the release step keeps a value under no named field"; mutant: "the release step names a field that could be text where it left it out"; mutant: "the release step writes an Accept it was not given"; mutant: "the release step keeps a header's name as sent" | +| A place left out is named only once its exchange is kept, and the file is written only in an absolute folder | `tests/test_release_exchanges.py::test_an_exchange_not_kept_leaves_no_place_named`, `tests/test_release_exchanges.py::test_it_writes_only_the_one_file_in_the_folder_the_workflow_names`, `release/exchanges.py`; mutant: "the release step names places of an exchange it did not keep"; mutant: "the release step writes into a relative folder" | diff --git a/examples/invalid/release-exchanges.a-redirects-address-kept.json b/examples/invalid/release-exchanges.a-redirects-address-kept.json new file mode 100644 index 000000000..a86644793 --- /dev/null +++ b/examples/invalid/release-exchanges.a-redirects-address-kept.json @@ -0,0 +1,86 @@ +{ + "schema": "release-exchanges/1", + "exchanges": [ + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory/git/matching-refs/tags/factory", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=utf-8", + "etag": "W/\"30eb71c9\"", + "location": "https://productionresultssa0.blob.core.windows.net/x" + }, + "body": [ + { + "ref": "refs/tags/factory/v0-next", + "object": { + "sha": "499835d5384ba81c466b17199edcde1848431630", + "type": "commit" + } + } + ] + } + }, + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory/actions/artifacts/9/zip", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/zip" + }, + "measured": { + "size": 1024, + "sha256": "14fbe23eefd818500379b102bccdce8df98bd560f317782f90a816a8da0cb2f4" + } + } + }, + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "PATCH /repos/combsmith/software-factory/git/refs/tags/factory/v0-next", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": { + "sha": "499835d5384ba81c466b17199edcde1848431630", + "force": true + } + }, + "response": { + "raised": "Unanswered" + } + } + ], + "removed": [], + "dropped": [ + "exchanges[0].response.body[0].node_id", + "exchanges[0].response.body[0].url" + ], + "unkept": 0 +} diff --git a/examples/release-exchanges.json b/examples/release-exchanges.json new file mode 100644 index 000000000..d6ee551a5 --- /dev/null +++ b/examples/release-exchanges.json @@ -0,0 +1,85 @@ +{ + "schema": "release-exchanges/1", + "exchanges": [ + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory/git/matching-refs/tags/factory", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=utf-8", + "etag": "W/\"30eb71c9\"" + }, + "body": [ + { + "ref": "refs/tags/factory/v0-next", + "object": { + "sha": "499835d5384ba81c466b17199edcde1848431630", + "type": "commit" + } + } + ] + } + }, + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory/actions/artifacts/9/zip", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/zip" + }, + "measured": { + "size": 1024, + "sha256": "14fbe23eefd818500379b102bccdce8df98bd560f317782f90a816a8da0cb2f4" + } + } + }, + { + "request": { + "as": { + "kind": "app", + "name": "release" + }, + "via": "rest", + "call": "PATCH /repos/combsmith/software-factory/git/refs/tags/factory/v0-next", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": { + "sha": "499835d5384ba81c466b17199edcde1848431630", + "force": true + } + }, + "response": { + "raised": "Unanswered" + } + } + ], + "removed": [], + "dropped": [ + "exchanges[0].response.body[0].node_id", + "exchanges[0].response.body[0].url" + ], + "unkept": 0 +} diff --git a/release/AGENTS.md b/release/AGENTS.md index d6d5804ff..582c7be77 100644 --- a/release/AGENTS.md +++ b/release/AGENTS.md @@ -10,8 +10,7 @@ Not built yet, by the stories that hold their rules: promote.py, cutover.py. | A Release is cut only from a verified commit of `main`, newer than every Release, of the Channel's major | `tests/test_release.py::test_each_rule_of_a_cut_refuses_with_its_own_code` | | `next` moves only through the eval gate, judged for that Release and read before anything is written | `tests/test_release.py::test_a_closed_eval_gate_refuses_every_cut_and_move_and_burns_no_version` | | A step moves only a Channel the ring file declares, to a commit one Release names alone | `tests/test_release.py::test_only_a_channel_the_ring_file_declares_is_moved`, `tests/test_release.py::test_a_channel_moves_only_to_a_commit_one_release_names_alone` | -| No step writes a Release tag | `tests/test_release.py::test_no_step_writes_a_release_tag_and_none_names_anything_but_a_channel` | -| No step moves `stable` until promotion is built | `tests/test_release.py::test_stable_is_never_moved_until_promotion_is_built` | +| No step writes a Release tag, or moves `stable` until promotion is built | `tests/test_release.py::test_no_step_writes_a_release_tag_and_none_names_anything_but_a_channel`, `tests/test_release.py::test_stable_is_never_moved_until_promotion_is_built` | | A rollback passes no gate and moves only back | `tests/test_release.py::test_a_rollback_passes_no_gate_and_never_moves_forward` | | By default it moves to the greatest Release below | `tests/test_release.py::test_a_rollback_by_default_takes_the_greatest_release_below_the_one_next_is_on` | | A commit's Release is its one Release tag; a Channel never is one, and two are none | `tests/test_release.py::test_a_commit_with_no_release_or_two_resolves_to_none` | @@ -38,3 +37,4 @@ Not built yet, by the stories that hold their rules: promote.py, cutover.py. | Only `.github/workflows/release.yml` runs the step, held to its shape: one job on `factory-release`, its token for this repo alone, no checkout an output names | `tests/test_release_workflow.py::test_each_counter_example_is_refused_for_its_one_fault` | | No other workflow names the release job's environment, key, token or step | `tests/test_release_workflow.py::test_only_release_yml_names_the_release_jobs_environment_key_token_or_step` | | A cut asked with no version, as the schedule asks one, is of the next patch of its major | `tests/test_release_workflow.py::test_a_cut_asked_with_no_version_is_of_the_next_patch_or_the_majors_first` | +| The step keeps its exchanges held to lists and redacted, never changing what it does | `tests/test_release_exchanges.py::test_the_leak_surface_is_left_out`, `tests/test_release_exchanges.py::test_a_step_that_stops_or_fails_still_keeps_what_it_sent` | diff --git a/release/exchanges.py b/release/exchanges.py new file mode 100644 index 000000000..3b6b64818 --- /dev/null +++ b/release/exchanges.py @@ -0,0 +1,159 @@ +"""What the release step sent through its token, and what GitHub answered, kept for its run (issue 531). + +The step's exchanges become an artifact of a public repo, which anyone signed in may download, so what is kept is held +to lists, and anything not on them is left out. `Recording` wraps the step's one transport and passes every request +through unchanged; for each it keeps the call, its Accept and the headers `tools/check_trials.py`'s `HEADERS` lets a +trial's fixture keep but `location`, a redirect's address, and of a JSON body only the fields `FIELDS` names, at any +depth. The token is never among them: +`Live` adds it below the request it is given, and `authorization` is no header a fixture keeps. A body that is not JSON, +an artifact's download, whose one redirect names a signed address, and a file read at a commit, whose text may be the +eval set's own, are kept as their size and their sha256 alone. Then `tools/redact_trial.py` takes out anything shaped +like a credential, as it does from a trial's capture, and says where. + +`write` puts it in the folder it is given, as NAME, a `release-exchanges/1`; nothing here prints it. Nothing kept may change what +the step does: a fault in the keeping drops that exchange, and is counted. +""" +from __future__ import annotations + +import base64 +import hashlib +import json +import pathlib +import re + +from tools import redact_trial +from tools.check_trials import HEADERS + +AS = {"kind": "app", "name": "release"} +NAME = "release-exchanges.json" # the one file it writes, which the workflow uploads +# The fields of a JSON body the step's reads and writes are judged by, at any depth, and none else: the refs and the +# writes (ref, object, sha, type, force), the comparison with main, the check runs and workflow runs of verify and the +# eval job, an artifact's listing, a Deployment and its status, and an error's message. +FIELDS = frozenset({ + "ref", "object", "sha", "type", "force", + "status", "ahead_by", "behind_by", "total_commits", "merge_base_commit", "base_commit", + "total_count", "check_runs", "check_suite", "app", "slug", "name", "conclusion", "head_sha", "started_at", + "completed_at", "id", "workflow_runs", "path", "event", "run_attempt", "created_at", "updated_at", "check_suite_id", + "artifacts", "size_in_bytes", "expired", "workflow_run", + "environment", "task", "description", "state", "auto_merge", "required_contexts", "transient_environment", + "production_environment", + "message", "documentation_url", +}) +# The headers kept: those a trial's fixture keeps, but a redirect's address, which can carry a signature. +KEPT_HEADERS = frozenset(HEADERS) - {"location"} +# A read whose body is a file's text at a commit, kept as its size and hash alone. +OPAQUE = re.compile(r"/repos/[^/]+/[^/]+/contents/") + + +def _measured(body) -> dict: + """A body by its size and its sha256, never its text: a text body as the bytes the transport gave it; a file read at + a commit, which GitHub answers as JSON holding it in base64, as the file's own bytes, so that it can be held to + the file at that commit; any other JSON as its fields serialized in order.""" + if isinstance(body, str): + raw = body.encode("utf-8", errors="surrogateescape") + elif isinstance(body, dict) and body.get("encoding") == "base64" and isinstance(body.get("content"), str): + raw = base64.b64decode(body["content"]) + else: + raw = json.dumps(body, sort_keys=True).encode("utf-8") + return {"size": len(raw), "sha256": hashlib.sha256(raw).hexdigest()} + + +GONE = object() # what _listed leaves out of a list + + +def _listed(value, dropped: list, where: str = "body", named: bool = False): + """`value` with only the fields FIELDS names, at any depth: a text, a number or a truth is kept only under such a + field, never alone or in a list no field names. Each place left out is added to `dropped`, by the name of the field + left out where that name is a plain identifier, and never by a name that could be text.""" + if isinstance(value, dict): + kept = {} + for key, item in value.items(): + if key in FIELDS: + kept[key] = _listed(item, dropped, "%s.%s" % (where, key), True) + else: + dropped.append("%s.%s" % (where, key) if key.isidentifier() else where + ".?") + return kept + if isinstance(value, list): + items = [_listed(item, dropped, "%s[%d]" % (where, i), named) for i, item in enumerate(value)] + return [item for item in items if item is not GONE] + if value is None or named: + return value + dropped.append(where) + return GONE + + +def _body(value, dropped: list, where: str): + """A JSON body as _listed keeps it; a bare value it leaves out is null.""" + kept = _listed(value, dropped, where) + return None if kept is GONE else kept + + +def _headers(given) -> dict: + return {name.strip().lower(): value for name, value in (given or {}).items() + if isinstance(name, str) and name.strip().lower() in KEPT_HEADERS and isinstance(value, str)} + + +def _opaque(req: dict) -> bool: + return bool(req.get("follow")) or bool(OPAQUE.match(req.get("path", ""))) + + +class Recording: + """The transport `inner`, which keeps each exchange it sends as the module says.""" + + def __init__(self, inner): + self.inner, self.exchanges, self.dropped, self.unkept = inner, [], [], 0 + + def send(self, req: dict) -> dict: + try: + answer = self.inner.send(req) + except Exception as err: + self._keep(req, None, type(err).__name__) + raise + self._keep(req, answer, None) + return answer + + def _keep(self, req: dict, answer, raised) -> None: + try: + here, gone = "exchanges[%d]" % len(self.exchanges), [] # what is left out, named once it is kept + headers = _headers(req.get("headers")) + if isinstance(req.get("accept"), str): + headers["accept"] = req["accept"] + sent = {"as": AS, "via": "rest", "call": "%s %s" % (req["method"], req["path"]), "headers": headers, + "body": _body(req.get("body"), gone, here + ".request.body")} + if answer is None: + self.exchanges.append({"request": sent, "response": {"raised": raised}}) + self.dropped += gone + return + body = answer.get("body") + if _opaque(req) or not isinstance(body, (dict, list, type(None))): + kept = {"status": answer.get("status"), "headers": _headers(answer.get("headers"))} + if body is not None: # an answer with no body has nothing to measure + kept["measured"] = _measured(body) + else: + kept = {"status": answer.get("status"), "headers": _headers(answer.get("headers")), + "body": _body(body, gone, here + ".response.body")} + self.exchanges.append({"request": sent, "response": kept}) + self.dropped += gone + except Exception: # what cannot be kept is left out, and counted; the step goes on as it would + self.unkept += 1 + + def kept(self) -> dict: + """What `write` writes, each credential-shaped string in it, a dropped field's name among them, taken out by the + trials' redaction.""" + return redact_trial.redact_json({"schema": "release-exchanges/1", "exchanges": self.exchanges, "removed": [], + "dropped": list(self.dropped), "unkept": self.unkept}, []) + + def write(self, folder) -> bool: + """Write `kept()` as NAME in `folder`, or nothing: True when written. It never raises, so it never changes the + step's outcome or what it prints. The folder is the one the workflow names: absolute and standing; a NAME there + already is never written over, as `open`'s "x" refuses it.""" + try: + where = pathlib.Path(folder) + if not where.is_absolute() or not where.is_dir(): + return False + text = json.dumps(self.kept(), indent=1, sort_keys=True) + with open(where / NAME, "x", encoding="utf-8") as out: + out.write(text + "\n") + return True + except Exception: + return False diff --git a/release/move.py b/release/move.py index f3e766d12..05f373cfd 100644 --- a/release/move.py +++ b/release/move.py @@ -18,7 +18,8 @@ no longer where the plan read it: a step run beside it, or a person, moved it, and the plan was made of another state (decision 0080). A hand move between that read and the write is not seen, since a ref's update compares nothing; and records a Deployment of each move of `next` once read back. A rollback reads no fact and passes no gate, so it works while the Factory is broken. It prints what it did, by codes alone, and exits 0, or 1 with the -code that stopped it. Nothing here reads a file at any tag. +code that stopped it. Nothing here reads a file at any tag. Where `RELEASE_EXCHANGES` names a folder, it keeps each +exchange in it as `release/exchanges.py` says, whatever stopped the step, and prints none of it (issue 531). The Channels a step may move are those `release/rings.json` declares, and `next` is the one a step moves. """ @@ -28,7 +29,7 @@ import pathlib import sys -from release import current, cut +from release import current, cut, exchanges ROOT = pathlib.Path(__file__).resolve().parent.parent RINGS = ROOT / "release" / "rings.json" @@ -160,10 +161,25 @@ def main(argv, stdin=None, rings=RINGS, environ=None, connect=None) -> int: if connect is None: from adapters.github import Live connect = Live + environ = dict(environ if environ is not None else {}) + # Where RELEASE_EXCHANGES names a folder, each exchange is kept in it for the run's artifact (issue 531), + # whatever stopped the step: kept as release/exchanges.py says, and never printed. + # One recording for the step, however many transports it makes, written even where it made none. + recording = exchanges.Recording(None) if environ.get("RELEASE_EXCHANGES") else None + + def made(token): + transport = connect(token) + if recording is None: + return transport + recording.inner = transport + return recording try: - done, refused = run(dict(environ if environ is not None else {}), connect, _json(rings)) + done, refused = run(environ, made, _json(rings)) except Exception: # one code, and no traceback, which could quote what GitHub answered done, refused = {"writes": [], "deployments": []}, "step_failed" + finally: + if recording is not None: + recording.write(environ["RELEASE_EXCHANGES"]) print(json.dumps(dict(done, refused=refused) if refused else done)) return 1 if refused else 0 if list(argv) != ["plan"]: diff --git a/schemas/release-exchanges.schema.json b/schemas/release-exchanges.schema.json new file mode 100644 index 000000000..01fc06586 --- /dev/null +++ b/schemas/release-exchanges.schema.json @@ -0,0 +1,298 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "What the release step sent through its token, and what GitHub answered, kept for its run's artifact (issue 531).", + "$comment": "Anyone signed in may download it, so what is kept is held to lists: the headers below, and of a JSON body the fields release/exchanges.py's FIELDS names, which no schema here can say at any depth; a download, a file at a commit and a body that is not JSON are kept as their size and sha256 alone. tests/test_release_exchanges.py holds what no schema can.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "exchanges", + "removed", + "dropped", + "unkept" + ], + "properties": { + "schema": { + "const": "release-exchanges/1" + }, + "exchanges": { + "type": "array", + "items": { + "$ref": "#/$defs/exchange" + } + }, + "removed": { + "description": "Where tools/redact_trial.py took out something shaped like a credential.", + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "dropped": { + "description": "Where a field no list names was left out.", + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "unkept": { + "description": "How many exchanges could not be kept, and were left out.", + "type": "integer", + "minimum": 0 + } + }, + "$defs": { + "exchange": { + "type": "object", + "additionalProperties": false, + "required": [ + "request", + "response" + ], + "properties": { + "request": { + "$ref": "#/$defs/request" + }, + "response": { + "$ref": "#/$defs/response" + } + } + }, + "request": { + "type": "object", + "additionalProperties": false, + "required": [ + "as", + "via", + "call", + "headers", + "body" + ], + "properties": { + "as": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "name" + ], + "properties": { + "kind": { + "const": "app" + }, + "name": { + "const": "release" + } + } + }, + "via": { + "const": "rest" + }, + "call": { + "type": "string", + "pattern": "^(GET|POST|PUT|PATCH|DELETE) /[^ ]*$" + }, + "headers": { + "type": "object", + "additionalProperties": false, + "description": "The headers a trial's fixture keeps (tools/check_trials.py's HEADERS), but location, a redirect's address: no other is written.", + "properties": { + "accept": { + "type": "string" + }, + "age": { + "type": "string" + }, + "cache-control": { + "type": "string" + }, + "content-length": { + "type": "string" + }, + "content-type": { + "type": "string" + }, + "date": { + "type": "string" + }, + "etag": { + "type": "string" + }, + "if-none-match": { + "type": "string" + }, + "last-modified": { + "type": "string" + }, + "link": { + "type": "string" + }, + "retry-after": { + "type": "string" + }, + "server": { + "type": "string" + }, + "user-agent": { + "type": "string" + }, + "vary": { + "type": "string" + }, + "x-accepted-github-permissions": { + "type": "string" + }, + "x-accepted-oauth-scopes": { + "type": "string" + }, + "x-github-api-version": { + "type": "string" + }, + "x-github-media-type": { + "type": "string" + }, + "x-github-request-id": { + "type": "string" + }, + "x-ratelimit-limit": { + "type": "string" + }, + "x-ratelimit-remaining": { + "type": "string" + }, + "x-ratelimit-reset": { + "type": "string" + }, + "x-ratelimit-resource": { + "type": "string" + }, + "x-ratelimit-used": { + "type": "string" + } + } + }, + "body": { + "description": "The JSON body sent, its fields held to FIELDS, or null." + } + } + }, + "response": { + "type": "object", + "additionalProperties": false, + "description": "What GitHub answered: a status, its headers and a body or its measure; or the transport's error, by its class alone.", + "properties": { + "status": { + "type": "integer", + "minimum": 100, + "maximum": 599 + }, + "headers": { + "type": "object", + "additionalProperties": false, + "description": "The headers a trial's fixture keeps (tools/check_trials.py's HEADERS), but location, a redirect's address: no other is written.", + "properties": { + "accept": { + "type": "string" + }, + "age": { + "type": "string" + }, + "cache-control": { + "type": "string" + }, + "content-length": { + "type": "string" + }, + "content-type": { + "type": "string" + }, + "date": { + "type": "string" + }, + "etag": { + "type": "string" + }, + "if-none-match": { + "type": "string" + }, + "last-modified": { + "type": "string" + }, + "link": { + "type": "string" + }, + "retry-after": { + "type": "string" + }, + "server": { + "type": "string" + }, + "user-agent": { + "type": "string" + }, + "vary": { + "type": "string" + }, + "x-accepted-github-permissions": { + "type": "string" + }, + "x-accepted-oauth-scopes": { + "type": "string" + }, + "x-github-api-version": { + "type": "string" + }, + "x-github-media-type": { + "type": "string" + }, + "x-github-request-id": { + "type": "string" + }, + "x-ratelimit-limit": { + "type": "string" + }, + "x-ratelimit-remaining": { + "type": "string" + }, + "x-ratelimit-reset": { + "type": "string" + }, + "x-ratelimit-resource": { + "type": "string" + }, + "x-ratelimit-used": { + "type": "string" + } + } + }, + "body": { + "description": "A JSON body, its fields held to FIELDS." + }, + "measured": { + "type": "object", + "additionalProperties": false, + "required": [ + "size", + "sha256" + ], + "properties": { + "size": { + "type": "integer", + "minimum": 0 + }, + "sha256": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + } + } + }, + "raised": { + "type": "string", + "pattern": "^[A-Za-z_][A-Za-z0-9_]*$" + } + } + } + } +} diff --git a/tests/test_release_exchanges.py b/tests/test_release_exchanges.py new file mode 100644 index 000000000..6972e7df1 --- /dev/null +++ b/tests/test_release_exchanges.py @@ -0,0 +1,259 @@ +"""The release step's own exchanges, kept for its run's artifact (issue 531): what `release/exchanges.py` keeps of each, +and what it leaves out, since anyone signed in may download an artifact of a public repo.""" +from __future__ import annotations + +import base64 +import contextlib +import hashlib +import io +import json +import os +import pathlib +import subprocess +import sys +import tempfile +import unittest + +from adapters import github +from release import exchanges, move +from tests.test_release_step import D, NEXT, SET, STORE, VERSION, cut_reads +from tests.test_tools import SHAPED +from tools.jsonschema_lite import validate + +ROOT = pathlib.Path(__file__).resolve().parent.parent +REPO = "combsmith/software-factory" + + +class Answering: + """A transport that answers each request with the next of `answers`, as `Live` would hand it back.""" + + def __init__(self, *answers): + self.answers, self.asked = list(answers), [] + + def send(self, req): + self.asked.append(req) + answer = self.answers.pop(0) + if isinstance(answer, Exception): + raise answer + return answer + + +def written(recording) -> str: + return json.dumps(recording.kept(), sort_keys=True) + + +class Kept(unittest.TestCase): + def test_the_leak_surface_is_left_out(self): + # Each of the five things box 1 of #531 names, sent as GitHub could, and none of it kept. + token = SHAPED["ghs"] + signed = SHAPED["signed url"] + recording = exchanges.Recording(Answering( + # A body's field no list names, beside one that is kept; and a header no fixture keeps, beside one it does. + {"status": 200, "headers": {"etag": "W/\"1\"", "set-cookie": "user_session=E2F1A08A3", "location": signed}, + "body": [{"ref": "refs/tags/" + NEXT, "object": {"sha": D, "type": "commit"}, "url": signed, + "node_id": "MDM6UmVm", "unknown": token}]}, + # An artifact's download, followed to the storage: its address and its bytes are never kept. + {"status": 200, "headers": {"content-type": "application/zip", "location": STORE}, + "body": "PK\x03\x04 the eval report, which may hold a held-out case"}, + # A file at a commit: its text, base64 in a JSON body, is never kept. + {"status": 200, "headers": {"content-type": "application/json"}, + "body": {"name": "manifest.json", "content": "c3ludGhldGljLWVhcmx5LXJldHVybg==", "sha": "f" * 40, + "download_url": signed}}, + # An error: its message and documentation are kept, redacted; anything else it carries is not. + {"status": 404, "headers": {}, "body": {"message": "Not Found " + token, "documentation_url": "https://docs", + "errors": [{"value": "the text of a file at a commit"}]}}, + # A download the storage answers as JSON is still kept by its measure alone, whatever its fields. + {"status": 200, "headers": {"content-type": "application/json"}, "body": {"message": "held-out JSON"}}, + # A body that is not JSON, at a path no rule names: its text is never kept. + {"status": 502, "headers": {"content-type": "text/html"}, "body": "a gateway's page, quoting a log"}, + )) + recording.send(github.request("GET", "/repos/%s/git/matching-refs/tags/factory" % REPO)) + recording.send(github.download("/repos/%s/actions/artifacts/9/zip" % REPO)) + recording.send(github.request("GET", "/repos/%s/contents/evals/manifest.json?ref=%s" % (REPO, D))) + recording.send(github.request("GET", "/repos/%s/compare/%s...%s" % (REPO, D, "e" * 40))) + recording.send(github.download("/repos/%s/actions/artifacts/10/zip" % REPO)) + recording.send(github.request("GET", "/repos/%s/git/ref/heads/main" % REPO)) + text = written(recording) + for left in (token, signed, STORE, "E2F1A08A3", "MDM6UmVm", "held-out", "c3ludGhldGlj", "text of a file", + "location", "gateway's page"): + self.assertNotIn(left, text, left) + kept = recording.kept()["exchanges"] + self.assertEqual(kept[0]["response"]["body"], [{"ref": "refs/tags/" + NEXT, + "object": {"sha": D, "type": "commit"}}]) + self.assertEqual(kept[0]["response"]["headers"], {"etag": "W/\"1\""}) + self.assertEqual(set(kept[1]["response"]), {"status", "headers", "measured"}) + self.assertEqual(set(kept[1]["response"]["measured"]), {"size", "sha256"}) + self.assertEqual(set(kept[2]["response"]["measured"]), {"size", "sha256"}) + self.assertEqual(kept[3]["response"]["body"], {"documentation_url": "https://docs"}) + self.assertEqual(set(kept[4]["response"]["measured"]), {"size", "sha256"}) + self.assertEqual(set(kept[5]["response"]["measured"]), {"size", "sha256"}) + self.assertIn("exchanges[3].response.body.message", recording.kept()["removed"]) + self.assertEqual(kept[0]["request"]["as"], {"kind": "app", "name": "release"}) + self.assertNotIn("authorization", json.dumps([e["request"]["headers"] for e in kept]).lower()) + + def test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named(self): + # A request's own body and headers are held as an answer's are, and every field left out, of either, is named + # in `dropped` by its place: the Claude review of #531. + recording = exchanges.Recording(Answering( + {"status": 201, "headers": {}, "body": {"ref": "refs/tags/x", "node_id": "MDM6", "url": "https://u"}}, + {"status": 204, "headers": {}, "body": None})) + recording.send({"method": "POST", "path": "/repos/%s/git/refs" % REPO, "accept": "application/json", + "headers": {"If-None-Match": "W/\"1\"", "X-Secret": "the secret"}, + "body": {"ref": "refs/tags/x", "sha": D, "payload": "what no list names"}}) + recording.send(github.download("/repos/%s/actions/artifacts/9/zip" % REPO)) + kept = recording.kept() + sent = kept["exchanges"][0]["request"] + self.assertEqual(sent["body"], {"ref": "refs/tags/x", "sha": D}) + self.assertEqual(sent["headers"], {"if-none-match": "W/\"1\"", "accept": "application/json"}) + self.assertEqual(sorted(kept["dropped"]), ["exchanges[0].request.body.payload", + "exchanges[0].response.body.node_id", + "exchanges[0].response.body.url"]) + # An opaque read with no body has nothing to measure, and is still a release-exchanges/1. + self.assertEqual(kept["exchanges"][1]["response"], {"status": 204, "headers": {}}) + schema = json.loads((ROOT / "schemas" / "release-exchanges.schema.json").read_text()) + self.assertEqual(validate(kept, schema), []) + + def test_a_value_is_kept_only_under_a_field_a_list_names(self): + # Muse's read of #531: a text in a list no field names, or a body that is a bare value, is no field, and so is + # left out; and a field's name that is not a plain identifier is never written where it was left out. + recording = exchanges.Recording(Answering( + {"status": 200, "headers": {" Content-Type ": "application/json"}, + "body": ["a held-out sentence", {"ref": "refs/tags/x"}, [["deeper"]], 7, None]}, + {"status": 200, "headers": {}, "body": {"held-out case synthetic-early-return": "v", "ref": "r"}})) + recording.send({"method": "POST", "path": "/repos/%s/x" % REPO, "headers": {}, "body": "a bare body"}) + recording.send(github.request("GET", "/repos/%s/git/ref/heads/main" % REPO)) + kept = recording.kept() + first = kept["exchanges"][0] + self.assertEqual(first["response"]["body"], [{"ref": "refs/tags/x"}, [[]], None]) + self.assertIsNone(first["request"]["body"]) + self.assertEqual(first["request"]["headers"], {}) # no Accept was given, so none is written + self.assertEqual(first["response"]["headers"], {"content-type": "application/json"}) + self.assertEqual(kept["exchanges"][1]["response"]["body"], {"ref": "r"}) + text = json.dumps(kept) + for left in ("held-out", "a bare body", "deeper"): + self.assertNotIn(left, text, left) + self.assertIn("exchanges[1].response.body.?", kept["dropped"]) + self.assertIn("exchanges[0].response.body[3]", kept["dropped"]) + + def test_an_exchange_not_kept_leaves_no_place_named(self): + # Argon's read of #531: what is left out is named only once its exchange is kept. + recording = exchanges.Recording(Answering( + {"status": 200, "headers": {}, "body": {"encoding": "base64", "content": "!!! not base64 !!!"}})) + recording.send({"method": "POST", "path": "/repos/%s/contents/f" % REPO, "headers": {}, + "body": {"unlisted": 1}}) + self.assertEqual((recording.kept()["exchanges"], recording.kept()["dropped"], recording.kept()["unkept"]), + ([], [], 1)) + + def test_a_file_at_a_commit_is_measured_as_the_file_itself(self): + text = b'{"schema": "eval-set/1"}\n' + recording = exchanges.Recording(Answering( + {"status": 200, "headers": {}, "body": {"name": "manifest.json", "encoding": "base64", + "content": base64.b64encode(text).decode()}})) + recording.send(github.request("GET", "/repos/%s/contents/evals/manifest.json?ref=%s" % (REPO, D))) + self.assertEqual(recording.kept()["exchanges"][0]["response"]["measured"], + {"size": len(text), "sha256": hashlib.sha256(text).hexdigest()}) + + def test_every_counter_example_of_the_trials_redaction_is_left_out_here(self): + # The Factory Builder's ruling on #531: one set of patterns, held to every case the trials' own tests use. + for name, value in SHAPED.items(): + recording = exchanges.Recording(Answering( + {"status": 200, "headers": {"etag": value}, "body": {"message": value, "ref": [value], "name": "x"}})) + recording.send(github.request("POST", "/repos/%s/git/refs" % REPO, body={"ref": value, "sha": D})) + self.assertNotIn(value, written(recording), name) + + def test_importing_the_redaction_starts_nothing_and_reads_nothing_but_modules(self): + # The ruling's first condition: what release/ imports of tools/ is pure at import. + probe = ("import sys\nseen = []\n" + "sys.addaudithook(lambda event, args: seen.append((event, args[0] if args else None)))\n" + "import release.exchanges\n" + "bad = [e for e in seen if e[0] in ('subprocess.Popen', 'os.system', 'os.exec', 'socket.connect'," + " 'os.spawn', 'os.posix_spawn') or (e[0] == 'open' and isinstance(e[1], str)" + " and not e[1].endswith(('.py', '.pyc')))]\nprint(bad)\n") + done = subprocess.run([sys.executable, "-c", probe], cwd=str(ROOT), capture_output=True, text=True, timeout=60) + self.assertEqual((done.returncode, done.stdout.strip()), (0, "[]"), done.stderr) + + def test_a_fault_in_keeping_never_changes_what_the_step_is_answered(self): + answer = {"status": 200, "headers": None, "body": {"ref": object()}} + recording = exchanges.Recording(Answering(answer, ValueError("refused"))) + self.assertIs(recording.send(github.request("GET", "/repos/%s" % REPO)), answer) + with self.assertRaises(ValueError): + recording.send(github.request("GET", "/repos/%s" % REPO)) + self.assertEqual(recording.kept()["exchanges"][-1]["response"], {"raised": "ValueError"}) + broken = exchanges.Recording(Answering("not an answer")) + self.assertEqual(broken.send(github.request("GET", "/repos/%s" % REPO)), "not an answer") + self.assertEqual((broken.kept()["exchanges"], broken.kept()["unkept"]), ([], 1)) + + def test_it_writes_only_the_one_file_in_the_folder_the_workflow_names(self): + folder = pathlib.Path(tempfile.mkdtemp()) + recording = exchanges.Recording(Answering()) + for wrong in ("relative", str(folder / "gone")): + self.assertFalse(recording.write(wrong), wrong) + # A relative folder that stands is refused too: it is wherever the step happens to stand (Muse, #531). + (folder / "here").mkdir() + before = os.getcwd() + try: + os.chdir(str(folder)) + self.assertFalse(recording.write("here")) + finally: + os.chdir(before) + self.assertEqual(list((folder / "here").iterdir()), []) + self.assertTrue(recording.write(str(folder))) + self.assertFalse(recording.write(str(folder))) # never over a file that stands + self.assertEqual(json.loads((folder / exchanges.NAME).read_text())["schema"], "release-exchanges/1") + + +class TheStep(unittest.TestCase): + def run_main(self, environ, connect) -> tuple: + out = io.StringIO() + with contextlib.redirect_stdout(out): + code = move.main([], environ=environ, connect=connect) + return code, out.getvalue() + + def test_a_cut_keeps_each_exchange_and_prints_what_it_printed_before(self): + folder = pathlib.Path(tempfile.mkdtemp()) + env = {"RELEASE_COMMAND": "cut", "RELEASE_VERSION": VERSION, "RELEASE_CUT_COMMIT": D, "RELEASE_TOKEN": "t"} + plain = self.run_main(env, lambda token: github.Replay(cut_reads())) + kept = self.run_main(dict(env, RELEASE_EXCHANGES=str(folder)), + lambda token: github.Replay(cut_reads())) + self.assertEqual(kept, plain) + self.assertEqual(plain[0], 0) + data = json.loads((folder / exchanges.NAME).read_text()) + schema = json.loads((ROOT / "schemas" / "release-exchanges.schema.json").read_text()) + self.assertEqual(validate(data, schema), []) + calls = [e["request"]["call"] for e in data["exchanges"]] + # The download and its storage's answer, two exchanges of the replay, are one send of the step. + self.assertEqual(len(calls), len(cut_reads()) - 1) + self.assertIn("POST /repos/%s/deployments" % REPO, calls) + text = json.dumps(data) + for left in (STORE, SET["cases"][0]["id"]): + self.assertNotIn(left, text) + self.assertNotIn("release-exchanges", plain[1]) + + def test_a_step_that_stops_or_fails_still_keeps_what_it_sent(self): + folder = pathlib.Path(tempfile.mkdtemp()) + env = {"RELEASE_COMMAND": "rollback", "RELEASE_TOKEN": "t", "RELEASE_TO": "factory/v0.1.0", + "RELEASE_EXCHANGES": str(folder)} + code, printed = self.run_main(env, lambda token: Answering(RecursionError("what GitHub answered"))) + self.assertEqual((code, json.loads(printed)), (1, {"writes": [], "deployments": [], "refused": "step_failed"})) + data = json.loads((folder / exchanges.NAME).read_text()) + self.assertEqual([e["response"] for e in data["exchanges"]], [{"raised": "RecursionError"}]) + self.assertNotIn("what GitHub answered", (folder / exchanges.NAME).read_text()) + + def test_a_step_that_stops_before_it_connects_still_writes_its_file(self): # Argon's read of #531 + folder = pathlib.Path(tempfile.mkdtemp()) + code, _ = self.run_main({"RELEASE_COMMAND": "promote", "RELEASE_EXCHANGES": str(folder)}, + lambda token: Answering()) + self.assertEqual(code, 1) + self.assertEqual(json.loads((folder / exchanges.NAME).read_text())["exchanges"], []) + + def test_with_no_file_named_nothing_is_kept(self): + folder = pathlib.Path(tempfile.mkdtemp()) + env = {"RELEASE_COMMAND": "cut", "RELEASE_VERSION": VERSION, "RELEASE_CUT_COMMIT": D, "RELEASE_TOKEN": "t"} + seen = [] + self.run_main(env, lambda token: seen.append(1) or github.Replay(cut_reads())) + self.assertEqual(list(folder.iterdir()), []) + self.assertEqual(seen, [1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_release_workflow.py b/tests/test_release_workflow.py index 5ed963722..e02a2539f 100644 --- a/tests/test_release_workflow.py +++ b/tests/test_release_workflow.py @@ -13,6 +13,7 @@ ROOT = pathlib.Path(__file__).resolve().parent.parent LIVE = (ROOT / ".github" / "workflows" / "release.yml").read_text() STEP_ENV = " RELEASE_TOKEN: ${{ steps.mint.outputs.token }}\n" +UPLOAD_STEP = " - if: always()\n uses: actions/upload-artifact@" def caught(problems: list, words: str) -> bool: @@ -101,7 +102,20 @@ def test_each_counter_example_is_refused_for_its_one_fault(self): "a with given a text": (LIVE.replace('\n with:\n', '\n with: x\n', 1), "has a with: it is not given"), # An empty `with:` on the step that takes none, and a step of text after the steps' keys (Argon's read of # #379). - "an empty with on a step that takes none": (LIVE + " with:\n", "its step step has exactly env, run"), + "an empty with on a step that takes none": (LIVE.replace(UPLOAD_STEP, " with:\n" + UPLOAD_STEP), + "its step step has exactly env, run"), + # Issue 531: the step's exchanges are uploaded whatever stopped it, as one artifact kept 90 days, and + # nothing prints them. + "the exchanges kept only when the step succeeds": (LIVE.replace(UPLOAD_STEP, UPLOAD_STEP.replace( + " - if: always()\n uses:", " - uses:")), "its upload step has exactly"), + "the exchanges printed": (LIVE.replace(UPLOAD_STEP, " - run: cat release-exchanges.json\n" + UPLOAD_STEP), + "its steps are mint, checkout, step, upload"), + "the exchanges under another name": (LIVE.replace("name: release-exchanges", "name: exchanges"), + "its upload step is given under with"), + "the exchanges kept 30 days": (LIVE.replace("retention-days: 90", "retention-days: 30"), + "its upload step is given under with"), + "a file of the exchanges not named": (LIVE.replace(" RELEASE_EXCHANGES: ${{ github.workspace }}\n", ""), + "its step step is given under env"), # A bare item beside the steps' mappings is not the narrow form at all (issue 371): it stands under no key # without a value, where a YAML reader could take it for more of the last step's command. "a step of text after the last step": (LIVE + " - justastring\n", "is not in the narrow form"), diff --git a/tests/test_tools.py b/tests/test_tools.py index 6665263b1..6227f096d 100644 --- a/tests/test_tools.py +++ b/tests/test_tools.py @@ -2309,6 +2309,17 @@ def test_compiled_code_is_refused_wherever_it_is(self): self.assertTrue(caught(problems, "kernel/ghost.pyc: compiled code; Python loads it ahead of, or instead of, " "the reviewed source, and no check reads it")) + def test_no_commit_holds_the_file_the_release_step_writes_at_the_root(self): # issue 531 + # The release job uploads release-exchanges.json from the checkout's root whatever stopped its step: one a + # commit carried, or a link, would be published in place of the step's own record. Anywhere else it is a name. + from release import exchanges + self.assertEqual(check_repo.EXCHANGES, exchanges.NAME) + root = tree({"release-exchanges.json": "{}\n", "docs/release-exchanges.json": "{}\n"}) + problems = check_repo.run(root) + self.assertTrue(caught(problems, "release-exchanges.json: where the release step writes its own exchanges")) + self.assertFalse(caught(problems, "docs/release-exchanges.json")) + self.assertTrue(caught(check_repo.run(tree({"Release-Exchanges.JSON": "{}\n"})), "Release-Exchanges.JSON: where")) + def test_committed_local_output_is_refused(self): # An unchecked-hash kernel/__pycache__/hidden.cpython-39.pyc replaces the reviewed kernel/hidden.py # at import, and `git add -f` commits it past .gitignore. What git holds is not local output. diff --git a/tools/check_examples.py b/tools/check_examples.py index 4342162d9..3ab50b47e 100644 --- a/tools/check_examples.py +++ b/tools/check_examples.py @@ -38,7 +38,7 @@ "config": "data", "hooks": "data", "lifecycle": "data", "authority": "data", "apps": "data", "rubric": "data", "acceptance": "data", "trial": "data", "role": "data", "feed": "data", "eval-case": "data", "eval-set": "data", "context-eval": "data", "controller": "data", "harness": "data", "rulesets": "data", "credential-ages": "data", "machine-user": "data", - "rings": "data", "release-fixture": "data"} + "rings": "data", "release-fixture": "data", "release-exchanges": "record"} RELEASE = "^factory/v(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)$" # What one Run carries, and what is written from it, holds codes, never text: a string in these schemas is held by an # enum, a const or a pattern anchored at both ends, unless it is a property named `body`, or sits under one named diff --git a/tools/check_repo.py b/tools/check_repo.py index 4e270b430..d7a23df33 100644 --- a/tools/check_repo.py +++ b/tools/check_repo.py @@ -117,6 +117,7 @@ def _stub(rel: pathlib.PurePath, tree) -> list: LINKS = set() # Compiled code. Python loads it ahead of the source beside it, or with none there, and no check reads it. COMPILED = {".pyc", ".pyo", ".so", ".pyd"} +EXCHANGES = "release-exchanges.json" # release/exchanges.py's NAME, at the checkout's root # The org's secrets baseline opens .gitignore, word for word (combsmith/.github, the # gitignore standard, "L1 — Secrets baseline"). When the org changes it, copy the new block in # and put its hash here; never edit between the markers. @@ -266,6 +267,11 @@ def _unloadable(rel: pathlib.PurePath) -> list: if walk.SKIP & set(folded[:-1]): problems.append("%s: in __pycache__/, Python's local output, committed; Python loads it ahead of the " "reviewed source, and no check reads it; remove it from git" % rel) + # The release step writes its exchanges at the checkout's root for the job's artifact (issue 531): one a commit + # carries would stand there first, and be uploaded publicly in their place, or, as a link, what it points at. + if folded == [EXCHANGES.casefold()]: + problems.append("%s: where the release step writes its own exchanges for its run's artifact; a committed one " + "would be uploaded in their place; remove it" % rel) if len(rel.parts) > 1 and folded[0] in walk.ROOT_ONLY: problems.append("%s: in corpus/, local output make_corpus.py writes, committed; nothing reads it from " "git, and the corpus is remade from the Kernel; remove it from git" % rel) diff --git a/tools/check_workflows.py b/tools/check_workflows.py index 30d2dceca..d49978aec 100644 --- a/tools/check_workflows.py +++ b/tools/check_workflows.py @@ -758,7 +758,9 @@ # one job, `release`, on its own environment, whose deployment branches are main's alone, so no other ref's copy of # this file reaches its key. Its token is minted by the one pinned action below, for this repo alone and the grants the # step needs; the checkout is the commit the run is of, never a ref an output names; and its one step is -# `python3 -m release.move`, given its inputs and the token under env: and nothing else. +# `python3 -m release.move`, given its inputs and the token under env: and nothing else. Its last step uploads the +# step's own exchanges, kept as release/exchanges.py keeps them, as the one artifact `release-exchanges`, whatever +# stopped the step, for 90 days as the eval job's are, and nothing prints them (issue 531). RELEASE_WORKFLOW, RELEASE_JOB, RELEASE_ENVIRONMENT = "release.yml", "release", "factory-release" RELEASE_TOP = {"name": "sf-release", "on": "", "permissions": "{}", "jobs": ""} # Its triggers, word for word: a list the reader keeps no entry of, an option or a schedule, is held this way alone. @@ -789,7 +791,8 @@ RELEASE_GRANTED = [("contents", "read")] # the job's own token: its writes are the App's # The one action that mints the release App's token, as CHECKOUT is the one checkout: a bump edits this too. APP_TOKEN = ("actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510", "# v3.3.0") -RELEASE_ORDER = ("mint", "checkout", "step") +RELEASE_ORDER = ("mint", "checkout", "step", "upload") +RELEASE_EXCHANGES = "${{ github.workspace }}" # where the step writes release-exchanges.json, uploaded from there RELEASE_STEP = "python3 -m release.move" RELEASE_GIVEN = { "mint": {"with": {"client-id": "${{ vars.RELEASE_APP_CLIENT_ID }}", "private-key": "${{ secrets.RELEASE_APP_KEY }}", @@ -799,9 +802,12 @@ "checkout": {"with": {"persist-credentials": "false"}}, "step": {"env": {"RELEASE_COMMAND": "${{ inputs.command || 'cut' }}", "RELEASE_VERSION": "${{ inputs.version }}", "RELEASE_TO": "${{ inputs.to }}", "RELEASE_CUT_COMMIT": "${{ github.sha }}", - "RELEASE_TOKEN": "${{ steps.mint.outputs.token }}"}}, + "RELEASE_TOKEN": "${{ steps.mint.outputs.token }}", "RELEASE_EXCHANGES": RELEASE_EXCHANGES}}, + "upload": {"with": {"name": "release-exchanges", "path": "release-exchanges.json", "if-no-files-found": "ignore", + "retention-days": "90"}}, } -RELEASE_STEP_KEYS = {"mint": {"id", "uses", "with"}, "checkout": {"uses", "with"}, "step": {"run", "env", "timeout-minutes"}} +RELEASE_STEP_KEYS = {"mint": {"id", "uses", "with"}, "checkout": {"uses", "with"}, "step": {"run", "env", "timeout-minutes"}, + "upload": {"if", "uses", "with"}} # What names the release job's environment, its key or its token, in a workflow that is not release.yml. RELEASE_ONLY = re.compile(r"factory-release|RELEASE_APP_|RELEASE_TOKEN|release\.move", re.I) EVAL_TOP = {"name": "factory", "on": "", "permissions": "{}", "jobs": ""} @@ -1519,11 +1525,14 @@ def eval_problems(text: str, root: pathlib.Path = ROOT) -> list: def _release_kind(step: dict): - """What one step of the release job is: the token's mint, the listed checkout, the release step, or None.""" + """What one step of the release job is: the token's mint, the listed checkout, the release step, the upload of its + exchanges, or None.""" uses = step.get("uses", {}).get("value", "") if "uses" in step else None if uses is not None and "run" not in step: if (uses, step["uses"]["comment"]) == CHECKOUT: return "checkout" + if (uses, step["uses"]["comment"]) == UPLOAD: + return "upload" return "mint" if (uses, step["uses"]["comment"]) == APP_TOKEN else None return "step" if _lines(step.get("run")) == [RELEASE_STEP] and "uses" not in step else None @@ -1572,7 +1581,8 @@ def release_problems(text: str) -> list: continue number = next(iter(step.values()))["step"] own = set(step) - RELEASE_STEP_KEYS[kind] - if own or (kind == "mint" and step.get("id", {}).get("value") != "mint"): + if own or (kind == "mint" and step.get("id", {}).get("value") != "mint") or \ + (kind == "upload" and step.get("if", {}).get("value") != "always()"): problems.append("its %s step has exactly %s" % (kind, ", ".join(sorted(RELEASE_STEP_KEYS[kind])))) for under in ("with", "env"): have = {e["key"]: e["value"] for e in entries if e["step"] == number diff --git a/tools/mutants.py b/tools/mutants.py index 52d2cf838..41987141b 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -12347,6 +12347,85 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("the Release's rules may name what reads or runs", 'tools/check_release.py', ' if isinstance(node, ast.Name) and node.id in FORBIDDEN_NAMES:', ' if False:', ('tests.test_release.Rings.test_the_release_rules_import_nothing_but_each_other',)), + # Issue 531: the release step keeps its own exchanges, held to lists, for its run's public artifact. + ('the release step keeps a field no list names', 'release/exchanges.py', + ' if key in FIELDS:\n', ' if True:\n', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ("the release step keeps a redirect's address", 'release/exchanges.py', + 'KEPT_HEADERS = frozenset(HEADERS) - {"location"}\n', 'KEPT_HEADERS = frozenset(HEADERS)\n', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ('the release step keeps a header no fixture keeps', 'release/exchanges.py', + 'name.strip().lower() in KEPT_HEADERS and', 'True and', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ("the release step keeps an artifact's download whole", 'release/exchanges.py', + ' return bool(req.get("follow")) or bool(OPAQUE.match(req.get("path", "")))\n', ' return bool(OPAQUE.match(req.get("path", "")))\n', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ('the release step keeps a file at a commit whole', 'release/exchanges.py', + ' return bool(req.get("follow")) or bool(OPAQUE.match(req.get("path", "")))\n', ' return bool(req.get("follow"))\n', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ('the release step keeps a body that is not JSON', 'release/exchanges.py', + ' if _opaque(req) or not isinstance(body, (dict, list, type(None))):\n', ' if _opaque(req):\n', + ('tests.test_release_exchanges.Kept.test_the_leak_surface_is_left_out',)), + ("the release step's exchanges are not redacted", 'release/exchanges.py', + ' return redact_trial.redact_json(', ' return (lambda kept, said: kept)(', + ('tests.test_release_exchanges.Kept.test_every_counter_example_of_the_trials_redaction_is_left_out_here',)), + ('a fault in keeping stops the release step', 'release/exchanges.py', + ' self.unkept += 1\n', ' raise\n', + ('tests.test_release_exchanges.Kept.test_a_fault_in_keeping_never_changes_what_the_step_is_answered',)), + ('the release step writes over a file that stands', 'release/exchanges.py', + 'with open(where / NAME, "x", encoding="utf-8") as out:', 'with open(where / NAME, "w", encoding="utf-8") as out:', + ('tests.test_release_exchanges.Kept.test_it_writes_only_the_one_file_in_the_folder_the_workflow_names',)), + ('the release step keeps nothing where a folder is named', 'release/move.py', + ' recording = exchanges.Recording(None) if environ.get("RELEASE_EXCHANGES") else None\n', ' recording = None\n', + ('tests.test_release_exchanges.TheStep.test_a_cut_keeps_each_exchange_and_prints_what_it_printed_before',)), + ('a release step that fails keeps nothing', 'release/move.py', + ' done, refused = {"writes": [], "deployments": []}, "step_failed"\n finally:\n', ' done, refused = {"writes": [], "deployments": []}, "step_failed"\n recording = None\n finally:\n', + ('tests.test_release_exchanges.TheStep.test_a_step_that_stops_or_fails_still_keeps_what_it_sent',)), + ('the release job uploads its exchanges only when the step succeeds', 'tools/check_workflows.py', + ' (kind == "upload" and step.get("if", {}).get("value") != "always()"):\n', ' False:\n', + ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + ('the release job keeps its exchanges 30 days', 'tools/check_workflows.py', + '"path": "release-exchanges.json", "if-no-files-found": "ignore",\n "retention-days": "90"}},', '"path": "release-exchanges.json", "if-no-files-found": "ignore",\n "retention-days": "30"}},', + ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + ("the release step's keeping starts a process at import", 'release/exchanges.py', + 'from tools import redact_trial\n', 'from tools import redact_trial\nimport subprocess; subprocess.run(["true"])\n', + ('tests.test_release_exchanges.Kept.test_importing_the_redaction_starts_nothing_and_reads_nothing_but_modules',)), + ('the release step keeps what it sends whole', 'release/exchanges.py', + '"body": _body(req.get("body"), gone, here + ".request.body")}', '"body": req.get("body")}', + ('tests.test_release_exchanges.Kept.test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named',)), + ('the release step keeps every header it sends', 'release/exchanges.py', + ' headers = _headers(req.get("headers"))\n', ' headers = dict(req.get("headers") or {})\n', + ('tests.test_release_exchanges.Kept.test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named',)), + ('a field the release step leaves out is not named', 'release/exchanges.py', + ' dropped.append("%s.%s" % (where, key) if key.isidentifier() else where + ".?")\n', ' pass\n', + ('tests.test_release_exchanges.Kept.test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named',)), + ('an answer with no body is measured as null', 'release/exchanges.py', + ' if body is not None: # an answer with no body has nothing to measure\n kept["measured"] = _measured(body)\n', ' kept["measured"] = None if body is None else _measured(body)\n', + ('tests.test_release_exchanges.Kept.test_what_is_sent_is_held_to_the_lists_too_and_each_field_left_out_is_named',)), + ("a file at a commit is measured as GitHub's envelope", 'release/exchanges.py', + ' elif isinstance(body, dict) and body.get("encoding") == "base64" and isinstance(body.get("content"), str):\n', ' elif False:\n', + ('tests.test_release_exchanges.Kept.test_a_file_at_a_commit_is_measured_as_the_file_itself',)), + ('a committed release-exchanges.json at the root passes', 'tools/check_repo.py', + ' if folded == [EXCHANGES.casefold()]:\n', ' if False:\n', + ('tests.test_tools.Repo.test_no_commit_holds_the_file_the_release_step_writes_at_the_root',)), + ('the release step keeps a value under no named field', 'release/exchanges.py', + ' if value is None or named:\n return value\n', ' if True:\n return value\n', + ('tests.test_release_exchanges.Kept.test_a_value_is_kept_only_under_a_field_a_list_names',)), + ('the release step names a field that could be text where it left it out', 'release/exchanges.py', + ' dropped.append("%s.%s" % (where, key) if key.isidentifier() else where + ".?")\n', ' dropped.append("%s.%s" % (where, key))\n', + ('tests.test_release_exchanges.Kept.test_a_value_is_kept_only_under_a_field_a_list_names',)), + ('the release step writes an Accept it was not given', 'release/exchanges.py', + ' if isinstance(req.get("accept"), str):\n headers["accept"] = req["accept"]\n', ' headers["accept"] = req.get("accept")\n', + ('tests.test_release_exchanges.Kept.test_a_value_is_kept_only_under_a_field_a_list_names',)), + ("the release step keeps a header's name as sent", 'release/exchanges.py', + ' return {name.strip().lower(): value for name, value in', ' return {name.lower(): value for name, value in', + ('tests.test_release_exchanges.Kept.test_a_value_is_kept_only_under_a_field_a_list_names',)), + ('the release step names places of an exchange it did not keep', 'release/exchanges.py', + ' here, gone = "exchanges[%d]" % len(self.exchanges), []', ' here, gone = "exchanges[%d]" % len(self.exchanges), self.dropped', + ('tests.test_release_exchanges.Kept.test_an_exchange_not_kept_leaves_no_place_named',)), + ('the release step writes into a relative folder', 'release/exchanges.py', + ' if not where.is_absolute() or not where.is_dir():\n', ' if not where.is_dir():\n', + ('tests.test_release_exchanges.Kept.test_it_writes_only_the_one_file_in_the_folder_the_workflow_names',)), ('a refused plan exits as a plan', 'release/move.py', ' return 1\n', ' return 0\n', ('tests.test_release.Plan.test_the_plan_is_printed_from_what_was_read_and_nothing_is_written',)), @@ -12651,11 +12730,11 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' return "mint" if (uses, step["uses"]["comment"]) == APP_TOKEN else None', ' return "mint"', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), ("a step's key more passes", 'tools/check_workflows.py', - ' if own or (kind == "mint" and step.get("id", {}).get("value") != "mint"):', - ' if kind == "mint" and step.get("id", {}).get("value") != "mint":', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + ' if own or (kind == "mint" and step.get("id", {}).get("value") != "mint") or \\\n', + ' if (kind == "mint" and step.get("id", {}).get("value") != "mint") or \\\n', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), ('the mint named otherwise passes', 'tools/check_workflows.py', - ' or (kind == "mint" and step.get("id", {}).get("value") != "mint"):', - ':', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + ' or (kind == "mint" and step.get("id", {}).get("value") != "mint") or \\\n', + ' or \\\n', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), ('a step given more passes', 'tools/check_workflows.py', ' if have != RELEASE_GIVEN[kind].get(under, {}):', ' if False:', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), @@ -12883,7 +12962,7 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("an empty current content supersedes", 'adapters/applied.py', 'if not isinstance(content_id, str) or not content_id:', 'if not isinstance(content_id, str):', ('tests.test_adapters.Recorded.test_a_later_review_by_the_same_lens_on_the_current_content_releases_a_thread_that_is_gone',)), ("the step step takes a with", 'tools/check_workflows.py', - '"step": {"run", "env", "timeout-minutes"}}', '"step": {"run", "env", "timeout-minutes", "with"}}', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + '"step": {"run", "env", "timeout-minutes"},\n', '"step": {"run", "env", "timeout-minutes", "with"},\n', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), ("a step of text after a step's keys is not read", 'tools/check_workflows.py', 'if lead < indent or (lead == indent and not raw.lstrip(" ").startswith("- ")):', 'if lead < indent or not raw.lstrip(" ").startswith("- "):', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), # Muse's read of #383: the one reader of the list refuses a pattern no path may hold. @@ -12907,8 +12986,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' found = controller.get("home") if isinstance(controller, dict) else None\n return found if isinstance(found, dict) else None\n', ' found = controller.get("home") if isinstance(controller, dict) else None\n return found\n', ('tests.test_release_step.Step.test_a_step_that_cannot_be_read_asks_nothing',)), ('a checkout at any pin is the release checkout', 'tools/check_workflows.py', - ' if (uses, step["uses"]["comment"]) == CHECKOUT:\n return "checkout"\n return "mint" if', - ' if "id" not in step:\n return "checkout"\n return "mint" if', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), + ' if (uses, step["uses"]["comment"]) == CHECKOUT:\n return "checkout"\n if (uses, step["uses"]["comment"]) == UPLOAD:\n return "upload"\n return "mint" if', + ' if "id" not in step:\n return "checkout"\n if (uses, step["uses"]["comment"]) == UPLOAD:\n return "upload"\n return "mint" if', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)), ('a release step that runs any command passes', 'tools/check_workflows.py', 'return "step" if _lines(step.get("run")) == [RELEASE_STEP] and "uses" not in step else None', 'return "step" if "run" in step and "uses" not in step else None', ('tests.test_release_workflow.Shape.test_each_counter_example_is_refused_for_its_one_fault',)),