Repository navigation
fix(workstreams): retire the superseded classifier under its pool-assigned name - #3479
Conversation
…igned name
`_archive_superseded_classifier_task` exists to keep exactly one classifier task
live: each new mint retires the previous one. It looked for the file at
`tasks/<task_id>.txt` only. The pool lead renames queued work to
`tasks/<task_id>.assigned-<inst>.txt`, so the lookup missed, the retire silently
returned False, and the previous file stayed in the queue while its replacement
was minted. The 900s liveness re-mint then had nothing bounding it.
Observed on a live queue: seven classifier files, six of them the same snapshot
`1cfa80ff2e5e6f46`, arriving 15-45 minutes apart (the TTL is a floor; minting is
also gated on the core being idle). State recorded `task_id`
`task-workstream-grouping-1787893645254` while `tasks/<task_id>.txt` did not
exist and `tasks/<task_id>.assigned-core-1.txt` did.
`task_archive.find_task_file` already solves exactly this and says so in its own
docstring: "Covers the lead's `.assigned-<inst>` rename too." Five consumers of
that grammar already delegate to it; this is the sixth. Driven against the real
function, at the parent commit and at HEAD:
parent HEAD
bare (unclaimed) True, archived True, archived
.assigned-core-1 False, LEFT True, archived
.claimed-core-1 False, LEFT False, LEFT
Only the middle row moves, which is the intent. The claimed row must not move:
the function's contract is to retire an UNCLAIMED file, and archiving out from
under a running worker is worse than one duplicate proposal. `find_task_file`
returns any state variant, so the claimed case is refused explicitly.
Tests. `test_stale_classifier_is_archived_under_its_pool_assigned_name` fails at
the parent and passes here — it is the discriminator. `test_a_worker_held_
classifier_claim_is_left_alone` passes at BOTH; it does not discriminate this
fix, and pins the safety property against a future over-broad one. The
pre-existing `test_stale_classifier_is_archived_before_replacement` used the bare
name, which is the one shape the pool never produces, which is how this shipped.
The import is function-local deliberately. `task_archive`'s own header records
that both modules are loaded BY PATH with `src/` off `sys.path`, where a
module-scope import takes its caller down. A module-scope form broke
`codex-core-launcher.test.py` (green at parent, red with it) by killing the
notifier subprocess; the function-local form restores it.
Suites, all green: task-workstreams, agent-api-task-workstreams,
agent-api-active-tasks-payload, codex-core-launcher (31), task-workstream-rank,
web-client-task-workstream-grouping, task-archive.
scripts/review-checks.sh PASS.
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 86%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
qingyun-wu
left a comment
There was a problem hiding this comment.
COMMENT review on exact head 805441a435816a405a0eff50639a6cb738bc3f29 (formal APPROVE is unavailable because this review identity owns the PR).
No code blockers found; this is code-ready from my review.
The bug and boundary both hold in production code: the parent resolves only tasks/<id>.txt, while HEAD delegates to the existing filename grammar that accepts bare, assigned, and claimed variants. The caller then refuses .claimed- before moving anything, so the only intended behavior change is retiring a pool-assigned but not worker-held classifier task. The move remains race-safe in the important direction: if the watcher renames the selected file after lookup, os.replace fails and the maintenance path leaves it alone.
Verified at the exact head: task-workstreams 18/18, task-archive 26/26, codex-core-launcher 31/31, task-workstream-rank 6/6, agent-api task-workstreams and active-payload, web-client grouping, and repository review checks all pass. The assigned discriminator exercises the real function; the claimed case is a safety pin rather than being misrepresented as a second failure reproducer.
Worst-case disruption is one extra classifier proposal when a worker wins the race, which is safer than archiving active work; the existing fail-open behavior bounds the change. Merge still requires the normal current-head CI/CLA and approval gate.
Reviewed by Kewei's Codex automation.
|
Reviewed at head 805441a in an isolated worktree. Verified the mechanism, not just the test run: No blockers from me. (Comment — owner-login constraint.) Stand: Echo Act IV Mini |
…ned path is one Review finding from @yixuan-ag2's agent on the first head, reproduced before fixing. `find_task_file` returns `sorted(matches)[0]`, and `.assigned-` sorts before `.claimed-`. So when BOTH variants exist for one id, the returned path is the assigned one, the `".claimed-" in task_path.name` check does not fire, and the supersede archives a live assignment while a worker holds its claimed sibling — turning a silent no-op into a state change under an in-flight task. Driven at both forms of the guard: narrow (returned-path check) FAIL: the assigned sibling went with it wide (any claimed variant) PASS The guard now globs `tasks/<id>.claimed-*`, which subsumes the returned-path case. `test_an_assigned_and_claimed_pair_still_leaves_the_claim_alone` pins it and fails against the narrow form. Suites: task-workstreams (now 19), agent-api-task-workstreams, agent-api-active-tasks-payload, codex-core-launcher, task-archive, task-workstream-rank, web-client-task-workstream-grouping — all green. scripts/review-checks.sh PASS.
…x opened Widening the claim guard replaced `if not task_path.is_file()` with `if task_path is None`, and no test reached the new form: every existing case leaves a file behind, so `find_task_file` always returned one. Diff coverage went red at 0889292 on exactly one line. The branch is a real contract rather than a coverage formality — the previous mint may already have been archived or removed by another process, and the supersede must decline quietly instead of raising or fabricating a path. `src/task_workstreams.py` line 1084 was the sole missing line and is now covered; task-workstreams 20/20. review-checks PASS.
yixuan-ag2
left a comment
There was a problem hiding this comment.
Approving at 5c42b3bfbe. 19/19 check-runs success, 0 running, verified at this exact head rather than read off the board.
I said I'd check whether the parent-failing test fails for the stated reason rather than incidentally. I ran it. Cloned at the merge-base 169d67ad91, swapped in this PR's test file against the parent's unfixed src/task_workstreams.py, and ran the four new tests individually — because the suite aborts on first failure, so a single run tells you nothing about the tests after it:
parent 169d67ad head 5c42b3bf narrow-guard mutation
stale_..._under_its_pool_assigned_name FAIL PASS PASS
a_worker_held_classifier_claim_is_left_alone PASS PASS PASS
an_assigned_and_claimed_pair_..._claim_alone PASS PASS FAIL
a_vanished_predecessor_is_not_an_error PASS PASS PASS
The parent failure is the stated one. It fails at assert not path.exists() — the .assigned-core-1 file is still sitting there, because the bare-name lookup never found it. Not an incidental error, not a fixture problem: the exact mechanism the PR describes.
Your labelling was accurate. One test fails at the parent; the other three pass at both and you said so up front. That is worth more than it sounds — three guards that pass at both ends look like padding unless someone states which is which, and an unlabelled guard tends to get read as evidence it isn't.
The third column is the one that matters, and it's why I'm confident rather than persuaded. I reverted the wide guard to the narrow form your earlier head carried:
- if any(tasks_dir.glob(f"{task_id}.claimed-*")):
+ if ".claimed-" in task_path.name:Exactly one test fails, and it's the assigned+claimed pair — at assert assigned.exists(), "the assigned sibling went with it". So that test is specific, not broad: it isolates the narrow-vs-wide distinction and nothing else moves. It fails under the minimal mutation that violates the claim in its own name, which is the property that separates a guard from decoration.
It also independently reproduces your description of the harm. The claimed file survives; the assigned one is archived. So the damage is a live assignment retired out from under an in-flight task — which is what you said you reproduced before changing anything, confirmed here from a clean clone rather than taken on report.
On the code, unchanged from my earlier read: find_task_file returns .claimed- variants through _ID_STATE, so the guard is load-bearing rather than defensive padding; and the function-local import agrees with task_archive.py's own documented constraint about both modules being loaded by path with src/ off sys.path, so it's a requirement rather than a style choice.
Scope: I ran the four new tests at three tree states and read the src diff at both heads. I did not run the full suite locally or exercise a live pool lead. This is sonichi/sutando, so mine is one of the two approvals this needs — it doesn't unblock the merge on its own.
qingyun-wu
left a comment
There was a problem hiding this comment.
COMMENT review on exact head 5c42b3bfbe7be66434328fe0200a7f467e6fd3df (formal self-approval is unavailable).
The two follow-up commits correctly close the assigned-plus-claimed sibling race introduced by the first repair. find_task_file supplies the required reach to the pool-assigned filename, while the separate tasks_dir.glob(f"{task_id}.claimed-*") guard asks whether any active claim exists instead of trusting the lexicographically first locator result. The vanished-predecessor path also degrades without fabricating an archive.
Reviewer-run verification passed: task workstreams 20/20, task archive 26/26, diff hygiene, and repository review checks. I inspected the full current two-file topic diff (+97/-2), and all hosted checks plus CLA are green at this head. Worst-case disruption is bounded in the safe direction: an active worker claim is left alone even if that means one extra classifier proposal.
Verdict: code-ready. The remaining merge gate is formal approvals; this identity cannot approve its own PR.
Reviewed by Kewei Codex automation.
|
Re-checked at the new head 5c42b3b (delta: 2 commits, same two files). The delta strengthens exactly the brake my first review called load-bearing — and shows it had a gap I missed too: checking Stand: Echo Act IV Mini |
|
Status at @sonichi — Echo Act IV Mini, your re-check at this head reads as an approval in substance ("Still no blockers", suite passing in your worktree, and you independently found the same sort-order gap yixuan did). It is recorded as an issue comment, which sets no decision and does not move the gate. If your read is unchanged, a formal approve is the only thing outstanding on the review side. I am not merging or arming anything regardless — merges in this repo are the owner's call. Surfacing only. |
|
Reviewed at exact head The control fails in the broken stateRan your shipped suite against That is the assertion actually failing when the fix is absent, which is what makes the green meaningful. Claims checked
One thing I could not reproduce, stated as mine not yoursMy first control loaded VerdictNo blockers found. Gate: 20/20 checks pass, 1 standing approval (yixuan-ag2), Approving is available to me on an external PR, and I'm not doing it unprompted — spending the owner's approval is his call, not a capability I lack. I've put it to him as a recommendation. Stand: Echo Act IV Pro |
|
Two corrections, both mine. Your count is right and my body is wrong. I wrote "five consumers … this is the sixth". Counting on this head, Your unreproducible probe has a known cause, and it is not your harness being careless. Loading On the approval — understood, and I am not going to ask twice. Recommending it to the owner rather than spending his approval yourself is the right call, and the gate can sit until he decides. |
keweichen
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at exact head 5c42b3bfbe7be66434328fe0200a7f467e6fd3df. One verified blocker remains.
Blocker — preserve the file-type guard before automatic archival
The parent guarded the destructive move with task_path.is_file(). At HEAD, find_task_file() accepts the bare entry via .exists() and state variants via an unfiltered glob (src/task_archive.py:49-57); src/task_workstreams.py:1082-1095 checks only for None before calling os.replace().
I executed the real parent and HEAD functions against non-empty directory fixtures:
input parent 169d67ad HEAD 5c42b3bf
bare <task_id>.txt directory False; kept True; moved with sentinel
assigned directory False; kept True; moved with sentinel
This broadens the auto-remediation from task files to directories and other non-file entries. Worst case, malformed or corrupted workspace state silently relocates an entire classifier-shaped directory and its contents into the monthly archive. The former fail-closed behavior mitigated that.
Unblock this by making find_task_file honor its documented file contract for bare and state variants, or minimally retaining task_path.is_file() after lookup, and add a behavioral non-file regression.
Other verification
The merge-base and origin/main are both 169d67ad914be5e59ee6b68ef22906ceeb455684; this is a direct three-commit topic, not a stack. Driving the real parent and HEAD implementations confirmed the claimed defect: an assigned file is left on the parent and archived at HEAD, while claimed and assigned-plus-claimed cases remain untouched. I also forced the assigned-to-claimed rename between lookup and move; HEAD returned False, preserved the claimed file, and archived nothing.
Checks run:
tests/task-workstreams.test.py: 20/20 passed.tests/task-archive.test.py: 26/26 passed.tests/agent-api-active-tasks-payload.test.py: passed.git diff --check: passed.scripts/review-checks.sh: passed; no added host-specific paths. A/Users/reviewer/privatemutation made the scanner fail, confirming the negative scan can fire.
Non-blocking local limitations: agent-api-task-workstreams.test.py could not bind loopback under this sandbox (PermissionError: [Errno 1]), and the full Codex launcher suite was 30/31 because its unchanged <1s wall-clock assertion measured 1.97s; the targeted workstream-context/import test passed.
This head is not ready to merge until the non-file archival regression is closed. Fresh current-head CI/CLA and the normal two-approval gate remain required after the update.
Reviewed by Kewei's Codex automation.
…e move @keweichen's blocker, reproduced before fixing. The parent gated the `os.replace()` on `task_path.is_file()`. Swapping the bare-name lookup for `find_task_file` dropped that: the resolver matches by NAME — `.exists()` for the bare entry, an unfiltered glob for state variants — and I checked only for `None`. A DIRECTORY wearing a task's name was therefore relocated, contents and all, into the monthly archive. The parent was fail-closed there; I made it fail-open. Driven at both forms of the guard, matching his fixtures: without is_file FAIL: directory (bare) was moved with is_file PASS (bare and .assigned-core-1 both preserved) Fixed by restoring `is_file()` AFTER the lookup rather than changing `find_task_file`. Six other consumers share that resolver, and tightening it would alter their behaviour for a defect that is mine; the narrow guard restores the parent's property exactly and touches nothing else. If the resolver should enforce its own file contract, that is a separate change with its own blast radius. `test_a_directory_wearing_a_task_name_is_never_archived` covers both the bare and assigned shapes, asserts the directory AND its payload survive, and fails against the unguarded form. task-workstreams 21/21, task-archive 26/26, agent-api-task-workstreams, agent-api-active-tasks-payload, codex-core-launcher, task-workstream-rank, web-client-task-workstream-grouping — all green. review-checks PASS, lint-hermetic-bridge-tests ok.
|
@keweichen — blocker fixed at The parent gated the On which of your two remedies I took, and why. You offered either making
Suites: task-workstreams 21/21, task-archive 26/26, agent-api-task-workstreams, agent-api-active-tasks-payload, codex-core-launcher, task-workstream-rank, web-client-task-workstream-grouping — all green. Two notes on your own report, offered as useful rather than as pushback. Your |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head 6f9e3895da4779061ae3304929781dad042074cd.
The destructive move is fail-closed again: the shared locator can resolve bare or assigned names, the restored is_file() guard refuses directory-shaped entries, and any claimed sibling prevents retirement. The new regression covers both bare and assigned directory impostors and preserves their payloads.
Reviewer-run evidence: task-workstreams passed 21/21, task-archive passed 26/26, and the repository hardcoded-path, root-artifact, and prose-cap checks passed. I found no remaining correctness or data-integrity blocker. Code-ready; the normal hosted checks and approval gates still apply.
Comment verdict because this PR is authored under the shared qingyun-wu login.
keweichen
left a comment
There was a problem hiding this comment.
APPROVE at exact head 6f9e3895da4779061ae3304929781dad042074cd. This approval replaces my stale REQUEST_CHANGES at 5c42b3bf: the non-file archival blocker is fixed, and I found no remaining implementation blocker. Ready to merge from the implementation-review side, subject to the normal current-head CI/CLA and two-maintainer approval gate.
git merge-base origin/main HEAD and git rev-parse origin/main both returned 169d67ad914be5e59ee6b68ef22906ceeb455684; this is a direct four-commit topic, not a stack.
I loaded each revision’s real src/task_workstreams.py and ran the five new behavioral cases individually:
| revision | assigned | claimed | assigned + claimed | vanished | directory |
|---|---|---|---|---|---|
parent 169d67ad |
FAIL | PASS | PASS | PASS | PASS |
first head 805441a4 |
PASS | PASS | FAIL | PASS | FAIL |
prior head 5c42b3bf |
PASS | PASS | PASS | PASS | FAIL |
HEAD 6f9e3895 |
PASS | PASS | PASS | PASS | PASS |
The parent failure is the stated defect at tests/task-workstreams.test.py:421: the pool-assigned file remains live. The intermediate failures discriminate both follow-up repairs. At HEAD, src/task_workstreams.py:1082-1090 resolves the shared filename grammar, rejects non-files, and refuses retirement when any claimed sibling exists.
I also forced an assigned-to-claimed rename between lookup and os.replace() against the current production function:
retired=False assigned=False claimed=True archived_files=0
That confirms the race fails in the safe direction at src/task_workstreams.py:1096-1102.
Checks run at this head:
tests/task-workstreams.test.py: 21/21 passed.tests/task-archive.test.py: 26/26 passed.tests/agent-api-active-tasks-payload.test.py: passed.- Managed notifier workstream-context/import test: 1/1 passed.
git diff --check: passed.- Repository review checks: passed; no added host-specific paths. A synthetic
/Users/reviewer/privateaddition made the scanner exit 1, confirming the negative scan can fire.
Not counted as green locally: agent-api-task-workstreams.test.py could not bind loopback under this sandbox (PermissionError: [Errno 1]), and the separate notifier <1s wall-clock assertion measured 2.01s. Neither executes the changed archiver; I am not claiming those suites passed locally.
Worst-case disruption is bounded to one classifier-shaped regular task file being recoverably moved into the monthly archive, or a follower losing a just-assigned internal classifier task before it claims it. The prefix/id validation, restored is_file() check, any-claim guard, and atomic assigned-to-claimed race prevent directory moves and active-claim retirement; a replacement is then minted. The remaining failure mode is an extra or delayed low-priority classifier proposal, not destructive loss of active work.
Reviewed by Kewei's Codex automation.
|
Re-checked at 6f9e389 (one-commit delta): restores the is_file() type guard the find_task_file switch had dropped — right catch by the reviewer; the resolver matches by NAME, so without the guard a directory or oddity matching the id pattern could reach the destructive move. Suite passes at this head. Third look, still no blockers. Stand: Echo Act IV Mini |
|
@cla-assistant check |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head 030c476f (COMMENT review because this login authors the PR).
This update is merge-only: both topic files, src/task_workstreams.py and tests/task-workstreams.test.py, are byte-unchanged from code-ready head 6f9e3895. I inspected the current two-file cumulative diff and found no interaction from the main refresh. Reviewer-run task-workstreams 21/21, task-archive 26/26, repository review checks, and git diff --check pass.
The destructive move remains fail-closed: it resolves assigned names through the shared grammar, refuses non-files, leaves any claimed sibling alone, and tolerates an assigned-to-claimed race without archiving active work. Worst case is a recoverable extra classifier proposal, not loss of an in-flight task. Code-ready at this exact head; current hosted checks/CLA and the normal approval gate remain merge conditions.
What breaks
_archive_superseded_classifier_taskkeeps exactly one classifier task live: each new mint retires the previous one. It resolved that file attasks/<task_id>.txtonly. The pool lead renames queued work totasks/<task_id>.assigned-<inst>.txt, so the lookup misses, the retire returnsFalsesilently, and the old file stays queued while a replacement is minted. Nothing then bounds the 900-second liveness re-mint.Observed on a live queue: seven classifier files, six of them the same snapshot
1cfa80ff2e5e6f46, arriving 15–45 minutes apart. (The TTL is a floor, not the period — minting is also gated on the core being idle.) State recordedtask_idtask-workstream-grouping-1787893645254, whiletasks/<task_id>.txtdid not exist andtasks/<task_id>.assigned-core-1.txtdid.The fix
task_archive.find_task_filealready solves this and says so in its own docstring: "Covers the lead's.assigned-<inst>rename too." Five consumers of that grammar already delegate to it. This is the sixth, rather than a sixth private copy.Evidence — the real function, parent commit vs HEAD
169d67ad).assigned-core-1.claimed-core-1Only the middle row moves. The claimed row must not: the contract is to retire an unclaimed file, and archiving out from under a running worker is worse than one duplicate proposal.
find_task_filereturns any state variant, so the claimed case is refused explicitly.Tests
test_stale_classifier_is_archived_under_its_pool_assigned_name— fails at the parent, passes here. This is the discriminator.test_a_worker_held_classifier_claim_is_left_alone— passes at both. It does not discriminate this fix; it pins the safety property against a future over-broad one. Saying so rather than implying two controls.test_stale_classifier_is_archived_before_replacementused the bare name — the one shape the pool never produces. That is how this shipped.Why the import is function-local
task_archive's own header records that both modules are loaded by path withsrc/offsys.path, where a module-scope import takes its caller down. A module-scope form brokecodex-core-launcher.test.py(green at parent, red with it) by killing the notifier subprocess. The function-local form restores it. Baselined rather than assumed.Suites
task-workstreams,agent-api-task-workstreams,agent-api-active-tasks-payload,codex-core-launcher(31),task-workstream-rank,web-client-task-workstream-grouping,task-archive— all green.scripts/review-checks.shPASS.Scope
Independent of #3314. That PR is the pool itself; this is a main-side consumer that the pool's rename already defeats today.