Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ Naming an opening Role in `config/defaults.json` is a one-line change, and the K
|---|---|
| A Role named for `open-pull-request` is refused until an adapter opens a pull request, a step of the Harness reaches it as CALLERS lists, and that step calls `opened`, each missing part named in turn; a read of the pulls, a write to them other than `POST`, a write to one pull request, a method or a path not written as text, a caller outside the Harness, a step reaching another writer, and a step calling no `opened`, another module's, or `harness.apply.opened` spelt out with `harness.apply` not imported do not count; a path held in a name bound once, a module's constant, a keyword and an f-string are read, a name bound twice and a pull request's own path are not; `opened` imported by its name, under another, relatively, or with `harness.apply` whole, does, and by its own name in `harness/apply.py` alone; none named passes; the counter-example is refused | `tests/test_tools.py::test_a_role_named_to_open_pull_requests_waits_on_its_writer`, `tools/check_authority.py`, `tools/check_examples.py`, mutant: "a Role named to open pull requests is never refused", mutant: "an opening Role named none is refused", mutant: "a read of a repo's pulls opens one", mutant: "a method not written as text may open one", mutant: "a write to one pull request opens one", mutant: "any request opens a pull request", mutant: "a caller outside the Harness reaches the opening", mutant: "a step reaching any writer reaches the opening", mutant: "a step need not record the opening", mutant: "any write to a repo's pulls opens one", mutant: "opened imported by its own name records nothing", mutant: "harness.apply.opened spelt out records the opening with nothing imported", mutant: "harness.apply imported whole records nothing", mutant: "a path held in a name is not read", mutant: "a name bound twice is read by its first", mutant: "a module's constant is not read", mutant: "an argument by keyword is not read", mutant: "an f-string is not read", mutant: "a relative import of the Harness records nothing", mutant: "opened by its own name records the opening anywhere", mutant: "opened by its own name records nothing in harness/apply.py", mutant: "any module's opened records the opening", mutant: "any call of a step's records the opening", mutant: "opened imported under another name records nothing", mutant: "harness.apply imported under another name records nothing", mutant: "the configuration is not held to the opening", mutant: "a counter-example configuration is not held to the opening", `examples/invalid/config.an-opening-role-with-no-writer.json` |
| What cannot be read stays red (the reviews of pull request 819): a name re-bound, augmented, deleted, or a loop's, a with's or a walrus's target, or a parameter shadowing a module's constant, is not read as bound once; `request` counts only as `adapters/github.py`'s, imported at the module's top by its name, under another or as its module's attribute, and bound by nothing else, so a helper of that name and another object's `request` do not; `opened`, `apply` or `harness` bound anywhere in the module but by its one import, by a parameter, an assignment, a def, a class, a later rebinding, a comprehension or a lambda, records nothing, nor `opened` in `harness/apply.py` without its own def; a path not formatted, formatted by two values, or with an f-string's conversion or format, is no repo's pulls; and (the second reviews) a name is read only from `name = ` in its scope's own body, never from a nested def, lambda, class or comprehension, a branch, a loop, a try or a with, and a module's value is read in the module; the value formatted in is a parameter given once, or its attribute, never a literal, an operator, a call, a tuple, a star or an f-string; an attribute or an item of `request`, its module, `opened`, `apply` or `harness` set or deleted rebinds that name; an opener decorated, or whose name is bound again in its module or class, is not the def; a module naming `globals`, `locals`, `vars`, `setattr`, `delattr`, `exec`, `eval`, `compile`, `__import__`, `importlib`, `modules`, `__dict__` or `__builtins__` is not read; a call in a nested def, lambda, class or comprehension, or after a return or a raise, is not made. The syntax cannot hold a request built and never sent, a parameter that holds a tuple, nor, as above, the calls in different flows. The marks are syntax, and assume code written in good faith: an `importlib` imported by a from-import, a call in a postponed annotation, an attribute set through another name for a module (`other = apply; other.opened = print`), and a method rebound in its class are of that kind; a crafted evasion is for the reviewers and the Maintainer's hand approval of the naming to catch | `tests/test_tools.py::test_a_role_named_to_open_pull_requests_waits_on_its_writer`, `tools/check_authority.py`, mutant: "a name is bound by assignment alone", mutant: "a name deleted is read as bound once", mutant: "a parameter binds nothing", mutant: "opened shadowed in the module records the opening", mutant: "harness.apply shadowed in the module records the opening", mutant: "harness shadowed in the module records the opening", mutant: "opened in harness/apply.py need not be its own def", mutant: "any callee named request builds a request", mutant: "any object's request builds a request", mutant: "adapters.github's request shadowed builds a request", mutant: "a path not formatted reads as formatted", mutant: "a literal holding the hole reads as formatted", mutant: "a path formatted by two values reads as formatted once", mutant: "an f-string's hole with a conversion or a format is read", mutant: "a name bound in a nested scope is read as the scope's", mutant: "a module's value is read in the function's scope", mutant: "any value formatted in is read", mutant: "an f-string's hole may hold any value", mutant: "a parameter re-bound is read as given", mutant: "a name bound once in the function is read as a parameter", mutant: "a variadic parameter is read as one value", mutant: "an attribute set replaces nothing", mutant: "a decorated def is read as its name's", mutant: "a def whose name is bound again is read as its name's", mutant: "an adapter that names what rebinds names unseen is read", mutant: "a step that names what rebinds names unseen is read", mutant: "a call in a nested scope is read as made", mutant: "a statement after a return is read as run", mutant: "a step's calls are read in every scope" |
| Every module of the Harness that CALLERS lists as reaching the opening records it, or reaches it only by calling a function of another such module that records it, as a step calls its job (Carbon's read of pull request 819, the Claude+security read, and the Factory Builder's call edge): one that does neither, or cannot be read, is named, and a module CALLERS does not list records nothing for one it does; importing a recorder is no call, so a module that imports `apply` and calls the writer itself, one that calls a writer's adapter as well as a recorder, one that calls a function of the job that records nothing, or names the recorder without calling it, and a job that opens and leaves the record to its step, are named; a call into a module that only calls a recorder is not read; `opened` from another module of the Harness, under an alias with a def of its own name called, or by a relative import in a module below `harness/`, where `.` is not `harness`, records nothing, nor `harness.apply` imported under another name, another root's `apply.opened` or another module's `opened`; a name bound twice is read by neither binding; a write to the pulls by `PUT` or `PATCH`, and a `POST` to one pull request's reviews, open none | `tests/test_tools.py::test_a_role_named_to_open_pull_requests_waits_on_its_writer`, `tools/check_authority.py`, mutant: "a step reaching the opening need not record it where another does", mutant: "a step that cannot be read is passed over", mutant: "a relative import below harness/ is read as from harness", mutant: "opened from any module of the Harness records the opening", mutant: "harness.apply imported under another name binds harness", mutant: "any root's apply.opened records the opening", mutant: "harness.anything.opened records the opening", mutant: "a name bound twice is read by its last", mutant: "any write to a repo's pulls opens one", mutant: "a step need not record the opening", mutant: "opened imported under another name records nothing", mutant: "only the first step reaching the opening need record it", mutant: "a step's call of a recorder is not read", mutant: "a step that calls the opening's adapter itself is read as calling a recorder alone", mutant: "a step that calls no recorder is read as calling one", mutant: "any function of a module reaching the opening is read as a recorder", mutant: "a recorder named without a call is read as called", mutant: "a step's relative import below harness/ is read as from harness", mutant: "a recorder's whole module is read as the function a step calls" |
Comment thread
don-petry marked this conversation as resolved.
Resolved
87 changes: 84 additions & 3 deletions tests/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -934,12 +934,12 @@ def test_a_role_named_to_open_pull_requests_waits_on_its_writer(self): # issue
opener = "adapters.opening:open_pull_request"
step = "harness/open_step.py"

def tree(adapter, harness, at=step, head="from adapters.github import request\n\n"):
def tree(adapter, harness, at=step, head="from adapters.github import request\n\n", more=None):
folder = tempfile.TemporaryDirectory()
self.addCleanup(folder.cleanup)
root = pathlib.Path(folder.name)
for part, text in (("adapters/opening.py", head + adapter), (at, harness)):
(root / part).parent.mkdir(exist_ok=True)
for part, text in [("adapters/opening.py", head + adapter), (at, harness)] + sorted((more or {}).items()):
(root / part).parent.mkdir(parents=True, exist_ok=True)
(root / part).write_text(text)
return root

Expand Down Expand Up @@ -1098,6 +1098,87 @@ def tree(adapter, harness, at=step, head="from adapters.github import request\n\
with self.subTest(harness=harness):
self.assertTrue(caught(check_authority.opening_problems(named, tree(opens, harness), callers),
"never call harness/apply.py's opened"))
# Carbon's read of PR 819. Every step that reaches the opening records it: a second that does not, or one that
# cannot be read, is named, and a module CALLERS does not list records nothing for a step it does.
bypass, silent = "harness/bypass.py", "def run(t):\n return None\n"
both = dict(callers, **{bypass: {opener}})
self.assertEqual(check_authority.opening_problems(named, tree(opens, records, more={bypass: records}), both), [])
for text in (silent, "def run(t:\n"):
with self.subTest(bypass=text):
problems = check_authority.opening_problems(named, tree(opens, records, more={bypass: text}), both)
self.assertTrue(caught(problems, "harness/bypass.py, which reach the opening, never call"), problems)
self.assertFalse(caught(problems, "open_step.py"), problems)
# A module that records nothing passes when it reaches the opening only by calling a function of another that
# records it, as a step calls its job (the Claude+security read; the Factory Builder's call edge). Importing a
# recorder is no call: a step that imports apply, or calls the writer itself as well, or a job's function that
# records nothing, or names the recorder without calling it, or a job that opens and leaves the record to its
# step, or a relative import below harness/, where `.` is not harness, is refused.
job, at_job = "harness/open.py", ("from adapters import opening\nfrom harness import apply\n\ndef job(t):\n"
" return apply.opened('e1', 1, 'b', opening.open_pull_request(t, 'r'))\n\n"
"def helper(t):\n return opening.open_pull_request(t, 'r')\n")
calls_job = "from harness import open as job_\n\ndef main(t):\n return job_.job(t)\n"
joined = dict(callers, **{job: {opener}})
self.assertEqual(check_authority.opening_problems(named, tree(opens, calls_job, more={job: at_job}), joined), [])
self.assertEqual(check_authority.opening_problems(named, tree(opens, calls_job.replace(
"from harness import open as job_", "from harness.open import job as run_job").replace("job_.job", "run_job"),
more={job: at_job}), joined), [])
own = "harness/apply.py"
for harness, more, listed in (
("from adapters import opening\nfrom harness import apply\n\ndef main(t):\n"
" return opening.open_pull_request(t, 'r')\n", {own: "def opened(*a):\n return a\n"},
dict(callers, **{own: {opener}})),
("from harness import apply\n\ndef main(t):\n return t\n", {own: "def opened(*a):\n return a\n"},
dict(callers, **{own: {opener}})),
("from adapters import opening\n" + calls_job.replace("return job_", "opening.open_pull_request(t, 'r')\n"
" return job_"), {job: at_job}, joined),
(calls_job.replace("job_.job(t)", "job_.helper(t)"), {job: at_job}, joined),
(calls_job.replace("job_.job(t)", "t(job_.job)"), {job: at_job}, joined),
("from harness import apply, open as job_\n\ndef main(t):\n return apply.opened(job_.job(t))\n",
{job: "from adapters import opening\n\ndef job(t):\n return opening.open_pull_request(t, 'r')\n"},
joined)):
with self.subTest(harness=harness):
self.assertTrue(caught(check_authority.opening_problems(named, tree(opens, harness, more=more), listed),
"which reach the opening, never call"))
below = "harness/sub/open_step.py"
self.assertTrue(caught(check_authority.opening_problems(
named, tree(opens, "from . import open as job_\n\ndef main(t):\n return job_.job(t)\n", below,
more={job: at_job}), {below: {opener}, job: {opener}}), "harness/sub/open_step.py, which reach"))
# The silent step sorted last is named too (Astra), not only the first.
last = "harness/zz_silent.py"
problems = check_authority.opening_problems(named, tree(opens, records, more={last: silent}),
dict(callers, **{last: {opener}}))
self.assertTrue(caught(problems, "harness/zz_silent.py, which reach the opening, never call"), problems)
self.assertTrue(caught(check_authority.opening_problems(named, tree(opens, silent, more={bypass: records}),
callers), "never call harness/apply.py's opened"))
# opened from another module of the Harness, under an alias with a def of its own name called, a relative import
# in a module below harness/, where `.` is not harness, harness.apply under another name with harness bound
# once by something else (a parameter, an assignment) or by nothing, another root's or another module's
# attribute, record nothing.
for harness, at in (("from harness.read import opened\n\ndef run(t):\n return opened(1)\n", step),
("from harness.apply import opened as rec\n\ndef opened(*a):\n return a\n\n"
"def run(t):\n return opened(1)\n", step),
("from . import apply\n\ndef run(t):\n return apply.opened(1)\n", "harness/sub/open_step.py"),
("from .apply import opened\n\ndef run(t):\n return opened(1)\n", "harness/sub/open_step.py"),
("import harness.apply as ap\n\ndef run(harness):\n return harness.apply.opened(1)\n", step),
("import harness.apply as ap\nharness = ap\n\ndef run(t):\n return harness.apply.opened(1)\n", step),
("import harness.apply as ap\n\ndef run(t):\n return harness.apply.opened(1)\n", step),
("import harness.apply\n\ndef run(other):\n return other.apply.opened(1)\n", step),
("import harness.apply\n\ndef run(t):\n return harness.read.opened(1)\n", step)):
with self.subTest(harness=harness, at=at):
self.assertTrue(caught(check_authority.opening_problems(named, tree(opens, harness, at), {at: {opener}}),
"never call harness/apply.py's opened"))
# A name bound twice is not read by its last binding either; a write to the pulls other than POST, and a POST to
# one pull request's reviews formatted with `%`, open none.
start = "def open_pull_request(transport, repo):\n "
for adapter in (start + "pulls = '/repos/%s/issues' % repo\n pulls = '/repos/%s/pulls' % repo\n"
" return request('POST', pulls)\n",
"PULLS = '/repos/%s/issues'\nPULLS = '/repos/%s/pulls'\n\n" + start + "return request('POST', PULLS % repo)\n",
start + "return request('PUT', '/repos/%s/pulls' % repo)\n",
start + "return request('PATCH', '/repos/%s/pulls' % repo)\n",
start + "return request('POST', '/repos/%s/pulls/1/reviews' % repo)\n"):
with self.subTest(adapter=adapter):
self.assertTrue(caught(check_authority.opening_problems(named, tree(adapter, records), callers),
"no adapter sends a write"))

def test_a_value_held_twice_must_agree(self):
self.table["max_rounds"] = 9
Expand Down
Loading
Loading