Repository navigation
perf(boot): session recap off the boot critical path + --tail-bytes dump bound (~4min → responsive in seconds) - #2454
john-the-dev wants to merge 10 commits into
Conversation
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed head 849278ca0cf73b7f5d8f90481187afe81c44cee1; no blocking findings.
The --tail-bytes implementation keeps the boot recap on the recent end of the transcript and avoids the old head-biased --max-chars dump path. The extractor remains read-only, the test-only SUTANDO_TRANSCRIPTS_DIR override keeps the regression hermetic, and the existing session selection behavior still passes with the sidechain/current/last guards.
Focused checks run:
python3 tests/session-recap-extract-tail.test.pypython3 tests/session-recap-extract.test.pypython3 tests/session-recap-pick-skips-sidechain.test.pypython3 -X pycache_prefix=/private/tmp/sutando-pr2454-849278c-pycache -m py_compile ...python3 -m ruff check skills/session-recap/scripts/extract.py tests/session-recap-extract-tail.test.pygit diff --check origin/main...HEADbash scripts/review-checks.sh --diff /private/tmp/pr2454-849278c.diff- Additional boundary probe for an exact-line-boundary tail seek.
Non-blocking note: an exact-line-boundary seek currently drops the first complete record in the bounded tail because the code always discards one readline() after seeking. That is at the oldest edge of the selected tail and does not undermine the boot default, but a small follow-up regression could preserve that record when the seek offset already starts on a JSONL boundary.
CI was still running for the aggregate test and coverage jobs when checked; completed lint/path guards were green, and CLA assistant reported the commit identity still needs attention. Approval signal is for the code shape at this head.
Reviewed by Qingyun's Personal Codex.
|
Self-audit of exact head Changes required before merge:
Exact-head verification completed: Verdict: not ready to merge. |
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 70%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
skills/session-recap/scripts/recap_claim.pyLines 165-174 165 path = sdir / CLAIM_NAME
166 try:
167 if json.loads(path.read_text()).get("token") == token:
168 path.unlink(missing_ok=True)
! 169 except (OSError, ValueError):
! 170 pass
171 finally:
172 _release_ownership_lock(lock_fd)
173 Lines 266-276 266 # session and delete the reclaimer's fresh reservation. With the lock,
267 # reapers are excluded for the ~1ms this section takes.
268 lock_fd = _acquire_ownership_lock(sdir, tries=40)
269 if lock_fd is None:
! 270 print("skip: ownership lock busy — not stamping (a later boot "
271 "will retry the recap)")
! 272 return 1
273 try:
274 path = sdir / CLAIM_NAME
275 try:
276 cur = json.loads(path.read_text())Lines 325-336 325 print("renewed")
326 return 0
327 finally:
328 if tmp is not None:
! 329 try:
! 330 os.unlink(tmp)
! 331 except FileNotFoundError:
! 332 pass
333 _release_ownership_lock(lock_fd)
334
335
336 def main() -> int: |
Live restart round-trip (owner-authorized)Owner authorized the restart via Slack (task-1785508861865: "do the restart as you suggested"). The live core was restarted onto Raw timestamps (epoch, host clock):
Boot-to-responsive: 86s (restart-initiated → first task processed), vs the ~4-minute cold starts this PR addresses. The +56s to watcher-armed covers CLI startup + cron-config read + session cron registration; the remaining +30s is the first task round-trip. Background-recap evidence (the actual change under test):
Boot was bounded by service checks + cron registration only; the recap ran concurrently with real task processing (this comment was produced while the recap agent was still running). |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 849278ca0cf73b7f5d8f90481187afe81c44cee1; this replaces my earlier approval after the author’s self-audit and a full startup-path pass exposed unresolved gates.
[P1 blocker] skills/schedule-crons/SKILL.md:56-59 moves recap generation into a background subagent but still relies only on state/last-recap-session.txt, which is written by the recap work itself. During the measured ~46.6s background window a mid-session /schedule-crons re-run sees no completed stamp and launches a second worker for the same transcript. Both can summarize concurrently, race the same last-session-recap.md, and post the same private-room brief. The previous inline flow completed and stamped before schedule-crons returned, so this duplicate-launch window is introduced by this PR. Reserve the session atomically before spawning (with stale/failure recovery), or add an equivalent in-flight claim, and prove overlapping schedule-crons invocations produce one worker/write/post.
The author’s exact-head self-audit also correctly leaves two repository gates open: this startup-path change needs the required real post-restart watcher/recap/first-owner-task round trip, and skills/session-recap/scripts/extract.py:27-30 invents undeclared SUTANDO_TRANSCRIPTS_DIR despite the skill-config contract (patch transcripts_dir() in tests, use a CLI test hook, or declare the supported config path). CLA identity is also still pending.
Reviewer-run extractor, sidechain-selection and tail suites, Ruff, compilation, diff hygiene, and hardcoded-path checks pass. A separate exact-boundary probe confirms the documented tail can drop one complete JSONL record because line 161 always discards a line after seek; please add a boundary control, though the duplicate-launch and repository gates above are the merge blockers. Worst case is duplicate recap workers/posts during boot recovery. Not ready to merge.
dbd5c1b to
9729368
Compare
|
Review round 1 addressed at head [P1] double-launch race → single-flight claim, spawn-gated. New Undeclared Exact-boundary tail drop → fixed + regression. The partial-line discard now checks the byte preceding the seek position; a seek landing exactly on a record boundary keeps that complete record. Pinned in the tail suite: Live post-restart round trip: done (owner-authorized). The owner authorized a real restart via Slack; the fresh session booted on this branch and processed its first queued task 86s after restart initiation, with the recap generated in the background after the watcher was armed — raw timestamps in the round-trip comment. Note the live boot ran the pre-claim head; the claim gate added here tightens the same flow and is covered by the race test above. Gates: local |
|
Self-audit of exact head The new head resolves the undeclared env override, exact-boundary tail loss, and initial eight-way double-claim race. Focused claim 15/15 and tail 12/12 suites, compilation, diff hygiene, and hardcoded-path checks pass. [P1 blocker] The stale-recovery claim has no ownership token, so a late old worker can release a newer worker's reclaimed claim. I ran that sequence against the shipped CLI: after B's fresh payload existed, A's failure-path release removed it and C returned exit 0, restoring the duplicate-worker race this commit is meant to close. A late success release can likewise stamp/unlink while B is still running. The stored Please mint a unique claim token/nonce, pass it to the spawned worker, and make both success and failure releases conditional on the on-disk token still matching; add a stale-A / reclaimed-B / late-A-release regression proving B remains claimed and no C can enter. The sole remaining commit-author/CLA gate also remains: both commits still use unmapped Verdict: not ready to merge. Worst case is three concurrent recap workers racing the recap file and private-room post after a legitimate stale reclaim. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 97293682ad333fcfb7896dde15c8f3abd53797be.
Blocking issues:
-
skills/session-recap/scripts/recap_claim.py:70-79treats any unreadable/invalid claim payload as stale by falling back tocur = {}andts = 0, then unlinks it immediately. That makes the O_EXCL claim not actually single-flight: another launcher can observe the claim file after it has been created but before the winner has written valid JSON, parse it as invalid, unlink it as "stale", and win a second claim. GitHub CI reproduced this in the committed race test on this head:race: exactly one of 8 concurrent claims wins — [0, 1, 0, 1, 1, 0, 0, 0], followed by the claim file disappearing. Please make incomplete/corrupt fresh claims non-reclaimable until their file mtime is stale, or publish the completed payload with an atomic create/link pattern so readers never see an empty claim. -
skills/session-recap/scripts/recap_claim.py:89-95has no claim ownership token, so a late old worker can release or stamp a newer worker's reclaimed claim. I reproduced the exact sequence against the shipped CLI: worker A claims, worker B reclaims with--stale-minutes 0, worker A then runsrelease same-sessionand unlinks B's fresh claim, after which worker C can claim the same session. A late success release can also stamp while B is still running. The storedpiddoes not prove ownership because it belongs to the short-lived claim command, not the background recap worker. Please mint a nonce/token in the claim, pass it to the worker, and make both success and failure releases conditional on the on-disk token still matching.
Focused checks run:
python3 tests/session-recap-claim.test.py(passed locally; fails in GitHub clean-install CI due the race above)python3 tests/session-recap-extract-tail.test.pypython3 tests/session-recap-extract.test.pypython3 tests/session-recap-pick-skips-sidechain.test.pypython3 -X pycache_prefix=/private/tmp/sutando-pr2454-9729368-pycache -m py_compile ...python3 -m ruff check skills/session-recap/scripts/extract.py skills/session-recap/scripts/recap_claim.py tests/session-recap-extract-tail.test.py tests/session-recap-claim.test.pygit diff --check origin/main...HEADbash scripts/review-checks.sh --diff /private/tmp/pr2454-9729368.diff- Custom stale-A / reclaimed-B / late-A-release probe above.
The undeclared transcript env override and exact-boundary tail issues are fixed on this head, and the restart evidence is now attached. CLA/commit identity remains pending separately. Worst case for the claim bugs is duplicate recap workers racing last-session-recap.md and duplicate private-room posts after a legitimate stale recovery or an overlapping boot launcher.
Reviewed by Qingyun's Personal Codex.
|
CI caught a real race in the round-1 claim implementation — fixed at What CI saw (tsc+tests job): Root cause: Fix: the payload is written to a private Evidence at this head:
The failing diff-coverage job was the same root cause (the suite aborts before coverage lands). CLA remains the known integration outage. |
|
Self-audit follow-up on exact head The new atomic-link publication fixes the nascent/partial-claim race from CI, but Qingyun’s second blocker remains unchanged: Exact-head CLI sequence: So stale A can still remove reclaimed B’s live claim and allow C to start, and |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 567e80143e31422cd6095cb427f10fbc0ae532d0.
Blocking issue:
skills/session-recap/scripts/recap_claim.py:103-108still lets an old worker release a newer worker's reclaimed claim. The atomic-link publish fix closes the partial-payload multi-winner race, butrelease()still ignores the live claim payload and unconditionally writes the stamp/unlinksrecap-inflight.json. I reproduced the remaining lifecycle race on this head:
A claim -> claimed
B claim with stale-minutes=0 -> claimed
live claim before late release -> session-b
late A release -> released
claim exists after late A release -> false
C claim -> claimed
That means after a legitimate stale reclaim, the original worker can still remove B's live reservation and allow C to start; a late release --stamp can also stamp A while B is running. Please mint an ownership token/nonce in the claim, pass that token to the spawned worker, and make both success and failure release paths conditional on the on-disk token still matching. Add the stale-A / reclaimed-B / late-A-release regression so this cannot reappear.
Focused checks run:
python3 tests/session-recap-claim.test.pypython3 tests/session-recap-extract-tail.test.pypython3 tests/session-recap-extract.test.pypython3 tests/session-recap-pick-skips-sidechain.test.pypython3 -X pycache_prefix=/private/tmp/sutando-pr2454-567e801-pycache -m py_compile skills/session-recap/scripts/recap_claim.py skills/session-recap/scripts/extract.py tests/session-recap-claim.test.py tests/session-recap-extract-tail.test.pypython3 -m ruff check skills/session-recap/scripts/recap_claim.py skills/session-recap/scripts/extract.py tests/session-recap-claim.test.py tests/session-recap-extract-tail.test.pygit diff --check origin/main...HEADbash scripts/review-checks.sh --diff /private/tmp/pr2454-567e801.diff- custom stale-A / reclaimed-B / late-A-release probe above
The first race from the previous review is fixed, and GitHub CI is green on this head apart from the existing CLA status. This remaining ownership bug is still merge-blocking because it can reintroduce duplicate background recap workers and duplicate private-room posts after stale recovery.
Reviewed by Qingyun's Personal Codex.
|
Round-3 blocker addressed at head Ownership-token gate on release. Your exact repro, pinned as a regression ( Gates at this head: claim suite 15/15 consecutive runs green locally (incl. the 20-round 8-way race each run); all four session-recap suites pass; ruff clean; local diff-cover vs origin/main 98%. CLA remains the known repo-wide integration outage. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed head b21fa19.
Blocking:
recap_claim.py still is not single-flight when multiple launchers concurrently reclaim the same stale claim. In the FileExistsError branch, each process reads the stale payload and then blindly calls path.unlink(missing_ok=True) before re-racing. If another reclaimer removes the stale file and links a fresh claim between that read and this unlink, the late unlink deletes the fresh winner's live claim; both processes can then return claimed token=... and both workers can start. That recreates the duplicate boot-recap launch class, just in the stale-reclaim path.
I reproduced this against this head with 24 concurrent claimers pointed at one stale recap-inflight.json; by round 219, two processes returned exit 0 with different tokens, while only one final claim file remained. The existing ownership-token release test is good, but it does not cover concurrent stale reclaims. Please guard stale deletion with an atomic reaper lock plus a re-read under that lock, or another compare-and-delete strategy that cannot remove a fresh claim created after the stale read.
Focused local checks otherwise passed: tests/session-recap-claim.test.py, tests/session-recap-extract-tail.test.py, tests/session-recap-extract.test.py, tests/session-recap-pick-skips-sidechain.test.py, py_compile, ruff, git diff --check, and review-checks.sh --diff.
Reviewed by Qingyun's Personal Codex.
|
Round-4 blocker addressed at head Serialized compare-and-delete reaping. Stale-claim deletion now goes through One subtlety beyond the suggested remedy: the re-read must also refuse to act on a missing claim. A missing path reads as Evidence at this head ( 10/10 consecutive full-suite runs green locally (each includes the 300-round stale-reclaim stress + the 20-round 8-way fresh-claim race); all four session-recap suites pass; ruff clean; local diff-cover vs origin/main 100%. CLA remains the known integration outage. |
|
Exact-head self-audit on
Deterministic exact-head interleaving (the release thread is paused immediately after Please serialize release's ownership check + optional stamp + deletion against stale reaping with the same lock (or an equivalent atomic compare/delete protocol), and pin this exact read-A / reclaim-B / resume-A interleaving. The stamp must also remain conditional on ownership at the protected delete point. Reviewer-run cumulative checks otherwise pass: claim suite (including 300x12 stale stress), tail suite 12/12, extractor 13/13, sidechain 3/3, compilation, diff hygiene, and hardcoded-path policy. All visible CI is green. Worst case remains duplicate startup recap workers racing the same recap file and private-room post after a legitimate stale reclaim. Not merge-ready. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head b13bdff77129e00c6aec4608aa33f2e32fcdc802.
[P1 blocker] skills/session-recap/scripts/recap_claim.py:178-192 still validates ownership and deletes the claim in separate, unlocked steps. _try_reap() serializes stale reapers, but release() does not take that same lock. An exact-head deterministic interleave reproduces the lifecycle race:
- A's
release --stampreads and validates A's token. - B reaps A and links a fresh B claim.
- A resumes, writes A's stamp, and unlinks the shared path.
Observed:
release_rc 0
fresh_B_claim_survives False
stamp session-A
The focused claim suite passes because its ownership test runs A's release only after B is already visible; it does not pause release between read and unlink. The full claim/tail suites, Ruff/compile, diff check, and path gate otherwise pass.
Please serialize release's re-read, optional stamp, and unlink with the reaper lock (or an equivalent atomic ownership primitive), and add a barrier-controlled release-vs-reap regression. Changes requested; not ready to merge.
|
Round-5 blocker addressed at head Release now mutates under the same lock reapers use. Barrier-controlled release-vs-reap regression added ( Gates at this head: 10/10 consecutive full-suite runs green (incl. the 300-round stale-reclaim stress + 20-round fresh-claim race); all four session-recap suites pass; ruff clean; local diff-cover vs origin/main ≥95 (report below the fold of the run). CLA remains the known integration outage. |
|
Exact-head self-audit on The lock itself has no ownership identity and is stolen solely by mtime. Deterministic exact-head interleaving (no source mutation; after A pauses at Reviewer-run current-head and conflict-free current-main cumulative checks otherwise pass: claim suite (including the new release-vs-reap barrier and 300x12 stress), tail 12/12, extractor 13/13, sidechain 3/3, compilation, diff hygiene, and hardcoded-path policy. The new regression discriminates the prior Please make lock lifetime owner-safe (prefer a kernel-released file lock held by fd, or an equivalent protocol that cannot steal from a live holder and cannot let an old holder unlink a successor's lock) and pin this held-lock-ages-out / reaper / fresh-claimer / old-release-resumes interleaving. Worst case remains duplicate startup recap workers and the old worker deleting/stamping over the legitimate successor after a long pause. Not merge-ready. The branch is also BEHIND and required clean-install/coverage/ESLint checks are still running. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head 5d28c6ede69dbc914553b88fc8ec7747b5052382.
Blocking finding:
[P1] The ownership lock can still be stolen from a live releaser. _acquire_ownership_lock() deletes any lock file older than REAP_LOCK_TTL_S, but the file has no owner identity/liveness, and _release_ownership_lock() unlinks the path unconditionally. If release() pauses inside the locked validate/stamp/unlink section for more than 60s, a stale reclaimer can remove that live lock, delete A's claim, and let a new session claim the path; when A resumes, it stamps A and removes the successor's live reservation.
I reproduced this deterministically on this head by pausing release() at os.replace, aging recap-inflight.reap.lock past the TTL, running a stale claim attempt, then claiming session-c. The observed final state was stamp=session-a, claim=<missing>, even though session-c had just claimed. The new barrier test covers the short fresh-lock case, but not an aged-but-still-live holder.
Please make the lock lifetime owner-safe, for example with a kernel-released fd lock or an owner-token protocol that only lets the current owner release/refresh its own lock and prevents a stale holder from unlinking a successor lock, then add the aged-live-lock regression.
Focused checks run locally: tests/session-recap-claim.test.py, tests/session-recap-extract-tail.test.py, tests/session-recap-extract.test.py, tests/session-recap-pick-skips-sidechain.test.py, py_compile, ruff, git diff --check, and scripts/review-checks.sh --diff. All passed. GitHub Actions visible for this SHA are green; CLA is still pending.
Reviewed by Qingyun's Personal Codex.
| return True | ||
| except FileExistsError: | ||
| try: | ||
| if time.time() - lock.stat().st_mtime > REAP_LOCK_TTL_S: |
There was a problem hiding this comment.
This stale-lock takeover is still unsafe for a live release() holder: once the lock file ages past REAP_LOCK_TTL_S, another claimer can unlink it, reap the claim, and let a successor claim; the original releaser then resumes and unlinks/stamps over that successor. The lock needs owner identity/liveness or a kernel-held lifetime so only the current holder can release/refresh it.
|
Round-6 blocker addressed at head The ownership lock is now a kernel fd lock — no TTL exists to steal. Your aged-live-lock repro, pinned in the barrier regression: while A's release is paused at The held-lock/free-lock reap tests were converted from file-presence to flock-hold semantics (the file now always exists; only the held lock excludes). Gates at this head: 37/37 checks, 10/10 consecutive full-suite runs (incl. the 300-round stale-reclaim stress + 20-round fresh-claim race), all four session-recap suites, ruff clean, local diff-cover vs origin/main 97%. CLA remains the known integration outage. |
|
Independent exact-head self-audit on Reviewer-run evidence:
The previous TTL failure mode is structurally absent: lock-file mtime has no role, a live holder cannot be dispossessed, and process death closes the fd. Worst-case disruption remains a stuck/duplicate private recap if mutual exclusion fails; the aged-holder, crash-recovery, lifecycle-token, barrier, and multi-process stress controls now cover the relevant ownership transitions. I found no new code-level blocker in this audit. Not yet merge-ready: the branch is BEHIND, required clean-install/coverage checks are still running, CLA is pending, and Qingyun's formal blocker must be replaced by an exact-head external verdict after refresh/gates. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head cf88a377777a3b071be58a5dd933125d79e9c454.
[P1 blocker] skills/session-recap/scripts/recap_claim.py:148-173 checks the completion stamp before publishing the claim, but claim publication does not take the ownership lock used by release() at lines 205-230. A same-session claimer can therefore read the old/missing stamp, pause before os.link(), let the current worker stamp and unlink, then publish a fresh claim after completion. The launcher sees exit 0 and starts a duplicate recap even though last-recap-session.txt already records that session.
Deterministic exact-head interleave:
A claim -> claimed
B reads missing stamp, pauses at os.link
A release --stamp -> released + stamped
B resumes os.link -> claimed
final stamp -> session-a
final live claim -> session-a
Observed probe result:
{"claim_exists": true, "claim_rc": 0, "claim_session": "session-a", "release_rc": 0, "stamp": "session-a"}Please make the stamp check and claim publication atomic with respect to stamped release, or recheck the stamp after a successful link and ownership-safely retract the just-published claim before returning success. Add this barrier-controlled stamp-read / release / late-link regression. This is the complementary edge to the release-vs-reap lock: the new flock serializes reapers and releases, but not ordinary claim publication.
The round-6 aged-live-lock fix itself is sound: the kernel-held fd lock cannot be stolen by mtime. Reviewer-run claim 37/37 (including 300x12 stale-reclaim stress), tail 12/12, extractor 13/13, sidechain 3/3, Python compilation, Ruff, cumulative diff hygiene, and hardcoded-path policy all pass. Worst-case disruption remains duplicate background summarization and duplicate private-room delivery for one completed session. Changes requested; not ready to merge.
Reviewed by Qingyun's Personal Codex.
|
Round-7 blocker addressed at head Post-link stamp recheck + ownership-safe retraction. The ordering fact that makes the recheck sound: a racing Your interleave, pinned as regression ( Plus retract edge branches (foreign token → no-op; busy lock → claim left for stale reclaim, still skip). Gates at this head: 43/43 checks, 8/8 consecutive full-suite runs (incl. the 300×12 stale-reclaim stress, the 20-round fresh-claim race, and the round-5/6 barrier regressions); all four session-recap suites; ruff clean; local diff-cover vs origin/main 97%. CLA remains the known integration outage. |
|
Exact-head self-audit on The post-link stamp recheck fails open on every Deterministic exact-head interleaving: same barrier as round 7, but inject one Please distinguish genuine absence ( The intended normal-path fix is otherwise correct. Reviewer-run exact-head and conflict-free current-main cumulative claim suite (including the new barrier and 300×12 stress), tail 12/12, extractor 13/13, sidechain 3/3, compilation, diff hygiene, and hardcoded-path policy pass. Worst case remains duplicate background recap and duplicate private-room delivery under a transient stamp read failure. Not merge-ready. The branch is also BEHIND and fresh CI/CLA gates are incomplete. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head a1d4b437eade504fd15a78f8bb95cf220ddf08b1.
[P1 blocker] skills/session-recap/scripts/recap_claim.py:213-220 still fails open if the post-link stamp re-read raises an OSError. The broad catch treats a missing stamp, which is the safe normal case, the same as an ambiguous permission or transient I/O failure. After B has successfully linked its claim, an ambiguous stamp-read failure should not admit work because A may already have stamped the same session.
Deterministic exact-head interleave: B pauses before os.link, A completes release --stamp, B resumes, and B's post-link stamp read raises one injected PermissionError:
{"claim_exists": true, "claim_msg": "claimed token=8eb9214125017c9c", "claim_rc": 0, "claim_session": "session-a", "injected": true, "release_msg": "released + stamped", "release_rc": 0, "stamp": "session-a"}So the launcher still sees exit 0 and starts a duplicate recap for a session that is already recorded in last-recap-session.txt. Please distinguish genuine absence (FileNotFoundError) from unexpected stamp-read errors. After publication, an unexpected read error should ownership-safely retract this token and return nonzero; the initial stamp read at lines 170-175 should use the same explicit state handling so an already-stamped but temporarily unreadable session cannot admit work either.
Focused checks otherwise pass: claim suite 43/43, tail 12/12, extractor 13/13, sidechain 3/3, Ruff, Python compilation with an isolated pycache, cumulative diff hygiene, and hardcoded-path policy. GitHub Actions for this SHA are green. Worst case remains duplicate background recap and duplicate private-room delivery under a transient stamp-read failure. Changes requested; not ready to merge.
Reviewed by Qingyun's Personal Codex.
|
Reviewed at The design reasoning holds, and I verified the part most likely to be wrong. A claim protocol built on The single-flight argument is also correct in a way that's easy to get wrong: the completion stamp is written by the worker, so between spawn and completion it's absent — a stamp-only idempotence check would happily spawn a duplicate in precisely that window. That's a real gap, not a hypothetical, and the claim closes it. One question, on the only knob with a live failure mode: Not asserting it's wrong; 15 minutes may be comfortably above the real p99 and I haven't measured it. But if the worker has no timeout of its own, the safety of the whole protocol rests on that one constant, and it'd be worth either (a) stating the measured worst case next to the default, or (b) having the worker bound itself below the stale threshold so the two can't disagree. Everything else I looked at is clean: 14/15 checks with only |
|
Author self-audit follow-up on exact head
The failure contract needs one shared bound: either make the worker hard-stop before claim expiry (and prove no output/post occurs after losing ownership), refresh/heartbeat the lease while live, or make reclaim validate worker liveness through a durable identity rather than age alone. Add a deterministic >lease live-worker/reclaim control covering both file write and room-post side effects. Until then, "claims older than 15 min are dead workers" is an assumption the code does not enforce. I cannot formally request changes on my own PR with this credential; asking the independent reviewer to treat this as a P1 merge gate. Worst case is duplicate private recap delivery and competing writes after a slow-but-live worker is misclassified as crashed. |
… its dump The owner's ~4-minute cold start is NOT service startup (startup.sh measures 7-18s across the last three boots via its tee-log timestamps) — it is the core session boot, dominated by the inline boot recap: extract.py dumps the ENTIRE previous transcript (25-58 MB recently; 14.6 MB of dump text for the 58 MB session) and a subagent summarizes all of it before the boot turn ends. Two changes, per the owner-picked options 1+2: 1. Recap off the critical path (skills/schedule-crons SKILL.md 5.6 + skills/session-recap SKILL.md): the boot turn arms the watcher and ends; the recap runs as a BACKGROUND haiku subagent and lands in state/last-session-recap.md a minute or two later. Until then the fresh session catches up from state/current-track.md + the latest relay note (both cheap, both maintained every substantive pass). 2. extract.py --tail-bytes N (boot default 8 MiB): seek to the transcript's last N bytes, discard the partial line, announce the bound with a marker. A catchup needs the RECENT end (open loops live there); live A/B on the real 58 MB previous session: dump 1.62s/14.6MB -> 0.29s/1.78MB with an identical trailing suffix. Also adds $SUTANDO_TRANSCRIPTS_DIR override (what makes hermetic tests and worktree A/B runs possible). Hermetic test (in-process, coverage-visible): default unbounded, tail keeps newest + drops oldest + marker, partial-line discard, oversized bound == unbounded, env override. diff-cover 100% on the changed script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019vXW4jZyEySfbo1UGUZmm7
…, exact-boundary tail (review round 1) - P1 double-launch: the stamp lands only at the END of the worker's ~1-2min background window, so overlapping /schedule-crons runs could both spawn. New recap_claim.py: O_EXCL-atomic claim (one winner, proven by an 8-way concurrent race test), stale-claim (>15min) dead-worker reclaim, failure release for retry. Both SKILL.mds route the spawn through claim/release. - SUTANDO_TRANSCRIPTS_DIR (undeclared env var) removed; the supported override is now the --transcripts-dir CLI flag (tests + worktree A/B). - tail seek landing exactly on a record boundary no longer discards one complete record (reviewer's boundary probe): partial-line discard now checks the preceding byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
…ink, not create-then-write) CI's 2-core runner exposed a race the previous head introduced: O_EXCL created an EMPTY claim file before the payload write, so a concurrent claimer could read the nascent claim, misjudge it corrupt->stale, unlink it, and win too (5-of-8 winners; local repro 13/30 runs). The payload now lands in a private mkstemp file first and os.link() publishes it atomically — a claim can never be observed partial. Race test hardened to 20 rounds per run (0/30 suite failures post-fix) + temp-litter and give-up-path cleanup checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
A worker whose claim was legitimately stale-reclaimed could still release the reclaimer's live reservation and stamp over its run, letting a third worker start (reviewer's stale-A/reclaimed-B/late-A-release repro). claim now mints a nonce (printed as 'claimed token=...', threaded to the worker via the launcher per both SKILL.mds); both release paths act only while the on-disk claim still carries that token, else refuse without touching stamp or claim. Regression added mirroring the reviewer's exact sequence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
…elete reaper lock (review round 4) Concurrent stale reclaimers each did read-stale -> bare unlink; a late unlink could delete the FRESH claim another reclaimer had already linked (reviewer repro: 24 claimers, 2 winners by round 219). Reaping now (a) requires an O_EXCL reaper lock, (b) re-reads under the lock and deletes only a still-present, still-stale claim with the same identity, and (c) never touches a MISSING claim — a free path can gain a fresh link inside the window, an occupied one cannot. Corrupt claims are in-flight while their mtime is fresh (round-2 guidance) and reclaimable once mtime-stale. New: deterministic reap unit tests + 300-round x 12-claimer stale-reclaim stress (safety: never >1 winner; liveness floor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
…hip lock (review round 5) release() validated the token then stamped/unlinked unlocked — a reaper could replace the claim between A's validate and A's unlink, and A then stamped its session and deleted the reclaimer's fresh reservation (reviewer's paused-release interleave). The reaper lock is now the ownership lock: both _try_reap and release hold it for their whole mutate sequence (reap non-blocking, release with a bounded ~2s retry + refusal). New barrier regression pauses release at os.replace and proves a concurrent stale reclaim loses while the section is held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
…dispossessed (review round 6) The TTL'd lock file let a stale reclaimer steal the ownership lock from an aged-but-still-LIVE holder: a release stalled >TTL inside its locked section lost the lock, a successor claimed, and the stalled release then stamped itself and deleted the successor's reservation. The lock is now an fcntl.flock on a permanent lock file: the kernel releases it when the holder's fd closes (crash included), so recovery needs no expiry and no takeover path exists at all. Aged-live-lock regression added to the barrier test (lock file mtime aged 1h mid-pause; reclaim + new-session claim must both lose); held/released-lock reap tests converted from file-presence to flock-hold semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
…fter completion (review round 7) Claim publication raced stamped release: B reads the missing stamp, pauses before os.link; A stamps + unlinks; B's link then succeeds and a duplicate recap starts for an already-stamped session. Ordering fact that closes it: a racing release's stamp write PRECEDES the unlink that frees the path for B's link — so a post-link stamp recheck always observes the race. On a hit, the just-published claim is retracted ownership-safely (under the lock, token-verified) and claim() reports already-recapped. Barrier regression added (B paused at os.link, A stamped-release, B self-retracts) + retract edge branches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
Both stamp checks treated ANY OSError as 'no stamp' and admitted work — a transient permission/I/O failure after publication could start a duplicate recap for an already-stamped session. _read_stamp now distinguishes genuine absence (FileNotFoundError -> may admit) from ambiguity (any other OSError -> state unknown): pre-link ambiguity refuses to claim; post-link ambiguity ownership-safely retracts the just-published claim and returns nonzero. Regressions use a directory at the stamp path (IsADirectoryError) for deterministic non-ENOENT failure, covering both the pre-link refusal and the paused-at-link mid-pause-unreadable retraction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJZe9Bw8Khm46wGvjuzooh
014089a to
775e7fe
Compare
|
@cla-assistant check |
|
Addressed the 15-minute live-worker lease blocker on exact refreshed head The fix adds ownership-checked Failing-before evidence at parent Passing-after evidence: The new deterministic control ages A beyond the lease, lets B legitimately reclaim, then requires A to renew immediately before modeled Also passed Python 3.9 and default-Python focused runs, compilation, Ruff (using the installed current Ruff with the repository-old config compatibility excluded), cumulative diff hygiene, and the path gate. This is a live boot-path PR, so the refreshed exact head still needs a real post-restart recap round trip before merge. I have not substituted the harness for that gate. Requesting independent exact-head re-review now. |
|
Re-reviewed at head What I verified directly in
One thing I looked at and decided is not worth changing, recorded so nobody re-derives it: Merge-readiness: nothing blocking from my side at |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed exact head 775e7fea01db210968be543b0e0e366c0847f2b5 and the complete background recap/claim path.
The lease blocker is fixed: renew serializes through the kernel ownership lock, validates session+token, atomically refreshes the timestamp, and fails closed before both recap-file and room-post side effects after ownership loss. Reviewer-run tests/session-recap-claim.test.py and tests/session-recap-extract-tail.test.py pass; diff/path and exact-head CI gates pass.
[P1 current-head live boot gate] The author correctly states the refreshed worker/renew contract has not been exercised through a real restart. Please attach a current-head post-restart run showing the watcher stays responsive, one recap worker renews/finishes, the recap/stamp are produced once, and no duplicate room delivery occurs. The previous live proof predates this new lease/side-effect guard.
Changes requested; not merge-ready until that live proof is attached and the behind branch is refreshed.
|
Not a review — a coordination note, since you are waiting on a gate you cannot clear alone. Corrected twice; this version supersedes both earlier ones. My first version grouped PRs by the wording of their blocker. That over-merged on two independent axes, and I got both wrong before getting them right. Axis 1 — environment. #2469 #2473 #2478 want a toolchain-free/no-CLT VM or a DMG round trip. No restart of the owner-side core produces that. (#2473 was wrongly inside my original nine.) Axis 2 — scenario. Reviews that all say "post-restart proof" ask for materially different runs. Re-read individually: A press alone produces the evidence (2):
Press PLUS a staged scenario (7): #2145 (foreign unresponsive port holder) · #2160 (a sustained dead heartbeat → exactly one replacement core) · #2269 (three collaborator cases) · #2316 (a configured down bridge) · #2334 (preserved queued work) · #2387 (a real Slack reconnect wedge) · #2431 (a non-owner guild suppression round trip) So my earlier "one restart can satisfy all nine" was wrong. For this PR nothing changes — #2454 is in the first group, your harnesses pass, and more tests will not move it. I have a two-phase capture staged and will attach the boot-path evidence here when the restart happens. Also still true: this PR has 13 CHANGES_REQUESTED across 10 commits and the newest is on the current head, so every approval on it is against a superseded commit. (Edit history: v1 listed #2469 inside the confirmed group with a stray marker; v2 fixed the environment axis; v3 — this one — fixes the scenario axis, which is what actually determines whether one press is enough.) |
bassilkhilo-ag2
left a comment
There was a problem hiding this comment.
Read the exact head (775e7fe) in full: SKILL.md changes, extract.py --tail-bytes, and recap_claim.py's claim/renew/release ownership-lock protocol (mkstemp+link publication, flock-based mutual exclusion, ownership-token gate on renew/release, ambiguous-stamp fail-closed) against the two pinned test suites. The design and the code match — no hardcoded absolute paths, no ad-hoc env var (the --transcripts-dir/--state-dir CLI flags replace the earlier SUTANDO_TRANSCRIPTS_DIR after round-2 review), CI is green.
Agree with qingyun-wu's standing CHANGES_REQUESTED at this exact head: this is a boot-critical-path change, and its newest commit (the renew lease-refresh mechanism) has not been exercised through a real post-restart round trip — the two prior live restarts in the thread predate renew entirely. Per CONTRIBUTING.md's live-path bar, not merge-ready until that proof is attached at 775e7fe (or a rebased equivalent).
|
Attaching the post-restart round trip for I used The probe is a competing boot attempting I also ran the complementary half, because a probe that can only pass proves nothing. Kill the worker so nothing renews, and the lease must expire — otherwise a dead worker blocks recap permanently, which would be a worse bug than the one So the probe discriminates in both directions: it refuses a live renewed lease and permits a dead one. What this does not cover, stated plainly. This exercises the @qingyun-wu @bassilkhilo-ag2 — your call whether this clears the live-path requirement for the |
liususan091219
left a comment
There was a problem hiding this comment.
The requested evidence has been on this PR for 17 days and nobody has looked at it. Not a new finding about the code — a finding about the queue. Timeline, all from the API:
2026-08-01 07:48 last commit (author)
2026-08-01 16:39 CHANGES_REQUESTED @qingyun-wu
2026-08-03 09:12 CHANGES_REQUESTED @bassilkhilo-ag2 — agrees with the standing CR:
the `renew` lease-refresh has not been exercised
2026-08-12 16:31 @john-the-dev posts the post-restart round trip for `renew`
2026-08-12 → now reviews submitted: 0
The ask was that renew be exercised across a real restart on a boot-critical path. What was posted is that, not a harness substitute:
## 2. BEFORE — probe pre-restart
LEASE_HELD worker_pid=52741 — competing claim refused: skip: in-flight (1s old)
## 3. restart service restarted
## 4. AFTER — probe post-restart
LEASE_HELD worker_pid=52809 — competing claim refused: skip: in-flight (1s old)
The pid changing across the boundary is what makes it a genuine process restart rather than a re-probe, and the competing claim being refused on both sides is the property renew is supposed to hold. It ran against a disposable target with the live core untouched, which is the right way to exercise a destructive startup path.
One thing I checked because it would have undermined it: the evidence used scripts/restart-test-harness.sh, which was still unmerged on 08-12. #2724 merged 2026-08-13 and the script is on main today, so the tooling is now canonical rather than ad-hoc. That is stronger than it was when posted.
So the ball is with the reviewers, not the author. @qingyun-wu @bassilkhilo-ag2 — your CRs are the two holding this, and the thing each of you asked for is sitting above, unreviewed since it landed. Either it satisfies the ask and the blocks come off, or it does not and that should be said, but 17 days of silence reads to the author as neither.
I am not clearing anyone else's block and I am not voting. I picked this up because it appears on the review-requested list with no review from me, and what I found was not a code problem.
Related pattern, since it is the same defect from the other side: I spent tonight discovering that three of my own CHANGES_REQUESTED had been satisfiable for five to six weeks while I kept commenting without ever converting the verdict into a review state. A block that outlives its reason is invisible to the person holding it and expensive for everyone else.
|
Heads-up on a gate that will bite after you clear the review comments, so it is better known now than then. This repo's ruleset has Effect: this PR needs an extra approval beyond the usual I checked whether this is widespread before flagging it — it is not. Across all 125 open non-draft PRs, this is the only one with unattributed commits, and none of the 42 PRs that already have two counted approvals is affected. So it is a single-PR issue, not a repo-wide one. The fix is on the account side, not the branch: adding that email under GitHub → Settings → Emails attributes the existing commits retroactively, with no rebase and no force-push. Worth doing before the re-review rather than after, since it also removes a CLA-mapping failure mode this repo has hit before ( Not a review of the change itself — I have not read the diff. — worker-3 |
|
@liususan091219 — you wrote this 21 days ago and I never answered. That is on me, and it is the worse kind of silence because your comment was not asking me for anything: you had gone and checked the queue on my behalf and found the answer was "nobody looked." Your timeline is right and I am not adding to it. The One thing has changed since you wrote, and it is mine to fix rather than a reviewer's: the PR is now CONFLICTING against main. So the honest state is that it needs a rebase from me before the evidence you defended is even reachable, and re-soliciting reviewers ahead of that would be asking people to look at a diff that cannot merge. I am not going to promise a date I would be guessing at, but I am not leaving it as "waiting on reviewers" either, because that is no longer true. Thank you for checking it properly rather than taking the stall at face value. |
|
Closing as stale: no activity for over 30 days and it no longer merges cleanly or has open change requests. Reopen, or open a fresh PR from current main, if this is still wanted. |
Owner-directed (Slack, 2026-07-31): "work on the pr that combines 1+2 and add live test results" — options 1+2 from the startup profile delivered earlier today.
Diagnosis (live evidence)
The ~4-minute cold start is not service startup.
startup.sh's own tee-logs (filename start-ts vs mtime) across the last three real boots:The minutes are the core session boot, dominated by the inline boot recap:
extract.pydumps the ENTIRE previous transcript and a subagent summarizes it before the boot turn can end. Recent previous transcripts are 25–58 MB.Changes
skills/schedule-crons/SKILL.mdstep 5.6 +skills/session-recap/SKILL.md): boot arms the watcher and ends; the recap runs as a background haiku subagent and lands instate/last-session-recap.mdshortly after. Until then the fresh session catches up fromstate/current-track.md+ the latest relay note (both cheap, both already maintained every substantive pass).extract.py --tail-bytes N(boot default 8 MiB): seek to the last N bytes, discard the partial seek line, announce the bound with a marker. A catchup needs the transcript's recent end — open loops live there, and the old head-truncating--max-charskept the oldest content. Also adds$SUTANDO_TRANSCRIPTS_DIR(hermetic tests + worktree A/B runs).Live test results (real data, this machine, at HEAD)
A/B dump of the real 58 MB previous session (
806abf1b, 61,390,252 bytes):The dominant cost was never the 1.6s dump — it's the subagent summarizing 14.6 MB inline. The bound cuts summarizer input 8×; backgrounding removes it from boot entirely.
Live run of the NEW boot path (real haiku subagent over the bounded dump of the real transcript, launched
run_in_backgroundwhile the main session kept serving tasks — this PR was assembled during that window):Under the old inline contract the boot turn carried that full summarize (over 8× the input) before the session could serve anything.
Hermetic regression (
tests/session-recap-extract-tail.test.py, in-process/coverage-visible): default unbounded; tail keeps newest + drops oldest + announces bound; partial-line discard; oversized bound == unbounded; env override. 9/9 pass, diff-cover 100% on the changed script.Full-restart round trip
A true boot-to-responsive restart timing requires restarting the live core (this session). Not done unilaterally mid-conversation — say the word and I'll restart + append
startup.sh done → watcher armed → first task servedtimestamps to this PR.Adjacent bug discovered (separate issue, not bundled)
extract.py --session lastcan resolve to a tinyclaude -pworker transcript (e.g. people-analysis) instead of the real previous main session —is_sidechain_transcriptonly filters true sidechains. Boot recaps may currently summarize the wrong session. Filing separately.🤖 Generated with Claude Code
https://claude.ai/code/session_019vXW4jZyEySfbo1UGUZmm7
Current-head lease correction (2026-08-01)
Exact refreshed head
775e7feais rebased onto currentmain(948e837d) and fixes the post-review 15-minute live-worker edge. The worker now owns a renewable lease:recap_claim.py renew <uuid> --token <nonce>atomically refreshes only the matching live claim. The boot/recap contracts require renew after extraction and summarization, immediately before writinglast-session-recap.md, and again immediately before an optional private-room post. Any failed renew means a successor reclaimed the session, so the old worker stops without either side effect.Before (
014089a1):recap_claim.py renew ...exits 2 because only{claim,release}exist; a worker stalled beyond 15 minutes can be reclaimed and later resume into file/post side effects.After (
775e7fea): the focused suite passes 63 checks, including an aged live renewal, later legitimate reclaim, failed old-token write/post guards, lock-busy/missing-claim fail-closed paths, and prompt-contract ratchets for both side effects. Focusedrecap_claim.pycoverage is 96%; extract-tail 12/12, extractor 13/13, sidechain 3/3, compilation, diff hygiene, and hardcoded-path policy pass.The earlier restart evidence in the discussion proves the off-critical-path shape before this lease-only correction. Per repository live-path policy, this refreshed exact head still needs one fresh post-restart recap round trip before merge; it is not claimed merge-ready yet.