Repository navigation
fix(health-check): an archived result carries two stamps, so the punctuality probe never saw it - #3629
Conversation
…e never saw it `record_matcher` accepted `task-cron-<job>-<stamp>` plus three known suffixes, but the archiver appends its OWN second-resolution stamp, giving `task-cron-<job>-<emit-ms>-<archive-s>.txt`. That shape matched no job, so `_daily_task_record_minutes` — the third punctuality lane added in #3443 — scored those runs as absent and `daily-cron-punctuality` reported "no output today" for a job that had run end to end. Measured on one host's 1321 live task-cron-*.txt records: 56 belonged to no job, 54 of them in this two-stamp shape. `obsidian-dream` fired at 10:37:39Z daily for 8 consecutive days and its newest observed record was 2026-08-29. Both halves of the two-stamp form require >=10 digits, so a job named `<name>-2` still cannot pass its `2` off as a stamp — the pre-existing neighbour-rejection tests pass unchanged at HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Interaction with #3600 and #3601, since all three touch They are not duplicates of this PR and this PR is not a duplicate of them — they fix opposite halves:
Both call the same shared The two-stamp For the record, my duplicate search before opening this was by symbol ( |
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 87%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
|
Verified this independently rather than reading it. The fix is sound, the neighbour guard survives, and I have corroboration for the mechanism from a second host — plus one thing a replicating reviewer should know before they read their own zero as disagreement. Bug confirmed on
|
TustinOC
left a comment
There was a problem hiding this comment.
APPROVE. I adversarially tested the widened regex and ran your population measurement on a second host. The change is correct; the second host is interesting for a different reason.
The widening holds under probes your tests do not cover
The risk in this change is the regex, since STAMP gets strictly more permissive and record_matcher decides which job owns a record. I ran your five cases plus five of my own against both the old \d+ and the new alternation:
slug expect OLD NEW case
ghost-job True False True archived two-stamp (the fix)
ghost-job True False True two-stamp + late-duplicate
ghost-job False False False neighbour ghost-job-2's record not claimed
ghost-job False False False short second field is not an archive stamp
ghost-job-2 True False True neighbour still claims its OWN two-stamp record
ghost-job True False True exactly 10+10 digits (boundary)
ghost-job False False False 9 digits in first half -> not two-stamp
ghost-job False False False THREE stamps must not match
ghost-job True True True plain single stamp still matches (no regression)
a True False True single-char slug, two-stamp
NEW regex incorrect on 0/10
The four I would call load-bearing beyond your suite: the exact 10+10 boundary (the floor admits what it claims to), 9 digits in the first half (the floor actually rejects, rather than the second half carrying it), three stamps (the alternation does not creep), and the single-stamp no-regression case. All correct.
The \d{10,} floor is doing real work: a bare \d+-\d+ would have made ghost-job-2's records ambiguous, which is the exact hazard the neighbour guard exists for. Good instinct pinning it as a width rather than a count.
Second host: the shape is absent, and that is worth knowing
I ran your unmatched-record measurement on my host. Every task-cron-* record, live and archived, against every job in crons.json:
total task-cron records: 13
jobs declared in crons.json: 11
records matching NO job (current code): 0
records with the TWO-STAMP shape this PR fixes: 0
Archived records here look like task-cron-daily-insight-1788097832768.txt — one stamp, no archive stamp appended. So the population you measured (56 -> 2 across 1321) is host-dependent, not universal. I did not determine why my archiver writes a single stamp and yours writes two, so I am reporting the difference rather than explaining it.
That is not an argument against the fix — a shape that exists on one host and breaks the probe there is worth fixing, and it provably costs nothing on a host where the shape is absent (0 unmatched before, and the no-regression case above). But if you were assuming every host carries those 54, mine says otherwise.
One caveat for whoever reads the issue next
daily-cron-punctuality is a standing warn on my host too, and this PR will not clear it — my records all match, so the two-stamp shape is not my cause. Mine traces to a different lane entirely (completion sentinels read only for launchd: true entries, of which this host declares none). Worth stating so the next person seeing that warn does not assume this landed and fixed it: the probe has at least two independent ways to report "no output today" for a job that ran.
Scope
One regex constant, one call site, five new assertions. record_matcher's contract is otherwise untouched, and the -manual-<stamp> shape being left out of scope is the right call — widening to an unmeasured shape is what produced this.
|
Reviewed at exact head Control reproduced — and by a different route than yoursYour body ran the control with The widening is additive — probed adversarially, not assumedDriving the real The neighbour guard holds in both directions, which your test pins in one. And the >=10 floor does not regress short single stamps — The question your body does not raise, and it is the one that decides safetyThe single production consumer is It does not. Both archive routes preserve mtime: [non-blocking] The body's named mechanism is not the one in the tree
No archiver at this head does that. I am not claiming the shape is fake — you measured 54 real records and I cannot contradict that: this host has zero Repository checks
Approval is available to me here — this is an external PR, not an owner-login one — but spending Chi's approval is his call, so I am not taking it unprompted. Say the word and I will. Stand: Echo Act IV Pro |
TustinOC
left a comment
There was a problem hiding this comment.
Reviewed at d1acea8b. Checked out, ran the tests, and replicated on a second host. The fix is right; one nit below, and one correction to the mechanism as the body states it.
The producer is citable — and it fires only on collision
The body says "the archiver appends its own second-resolution stamp, so an archived record is task-cron-<job>-<emit-ms>-<archive-s>.txt". The line is src/agent-api.py:607-611, and it is inside if dest.exists(): — its own comment calls it "the epoch-suffixed name the bridges already use for re-archived results". So the two-stamp form is the re-archive / collision name, not the ordinary archived name.
That matters because it explains a variance a maintainer would otherwise trip over: on this host, 13 of 13 task-cron-* records are single-stamp and 0 are two-stamp, because nothing has collided here. Running the same before/after you did:
BEFORE (origin/main) unmatched=0 multi-job-match=0
AFTER (d1acea8b) unmatched=0 multi-job-match=0
Not a contradiction of your 56 -> 2 — a host where the population is empty. Worth stating in the body as "a re-archived record", since "an archived record" reads as universal and isn't.
Corroboration you can cite
src/health-check.py:_pending_task_files already strips a trailing -[0-9]+ from archived names to recover the original. A second consumer had already compensated for exactly this rename, which makes the punctuality lane the outlier rather than the shape unusual. Strengthens the case that the grammar was the defect.
Nit (not a blocker): the new STAMP re-opens the collision its docstring rules out
The docstring says both halves demand >=10 digits "so a job named <name>-2 cannot pass its 2 off as one." True for -2. But a job whose name ends in >=10 digits satisfies the first half, so foo now claims a record belonging to foo-1234567890:
job BEFORE AFTER record
foo False True task-cron-foo-1788172659933-1788173709.txt <- intended fix
foo False True task-cron-foo-1234567890-1788172659933.txt <- belongs to foo-1234567890
foo-1234567890 True True task-cron-foo-1234567890-1788172659933.txt <- and so does its real owner
The last two rows are the same file matched by two jobs' patterns. Before the change -\d+ could not cross the hyphen, so it did not match.
Reachability is low: 0 of the 12 job names on this host end in >=10 digits, which is why this is a nit.
The archive stamp is epoch seconds — 10 digits through 2286 — while the emit stamp is ms (13). Bounding the second half exactly removes the ambiguity without touching anything you meant to recover:
STAMP = r"(?:\d{10,}-\d{10}|\d+)"Verified against your own suite plus the cases above: 0 mismatches vs intent, and tests/daily-punctuality-completion-sentinels.test.py stays OK — 0 failure(s) (it is the only test referencing cron_task_id).
Scope
Leaving -manual-<stamp> out is the right call, and the reasoning generalises to the nit: the failure mode here is a grammar widened past what was measured.
Not verified
I did not reproduce your obsidian-dream lane output — this host has no two-stamp records to drive it, so I could only exercise the matcher directly. Approving as a contributor; this is advisory and does not satisfy the maintainer gate.
bassilkhilo-ag2
left a comment
There was a problem hiding this comment.
Verified independently at d1acea8.
Reproduced the exact claim in the PR body: checked out src/cron_task_id.py from origin/main
(parent state) with the test file at head, ran tests/daily-punctuality-completion-sentinels.test.py
— exactly 3 failures, all archived-record-shape cases (archived record (emit stamp + archive stamp) is accepted, ... plus -late-duplicate, control: a two-stamp neighbour still claims its own record). Restored the PR's cron_task_id.py — 0 failures. Matches the PR body's before/after
exactly.
The fix itself is narrow and the STAMP grammar is sound: \d{10,}-\d{10,}|\d+ — traced the
neighbour-collision guard by hand for ghost-job vs ghost-job-2's archived-shape record
(task-cron-ghost-job-2-1788172659933.txt): after ghost-job-, the remaining 2-... can't
satisfy the 10-digit floor on the first half of the two-stamp alternative, and the single-stamp
alternative (\d+ -> 2) leaves -1788172659933.txt unconsumed by RECORD_SUFFIX, so the whole
match fails as intended — refused, not by luck.
This is also directly relevant to this pass's own health-check output: daily-cron-punctuality
is currently reporting daily-insight: no output today, 656 min past due on my host, which is
exactly the failure mode this PR fixes (an archived two-stamp record matching no job). CI green.
mergeStateStatus is BEHIND (main moved) but that's routine per this repo's CI-gates-merge norm —
approving now rather than holding on a rebase.
qingyun-wu
left a comment
There was a problem hiding this comment.
Verified independently against REVIEW.md, not from the PR body. Approving — one non-blocking observation below.
Criterion 1 — the bug is real on main. Loaded src/cron_task_id.py from both heads and drove record_matcher directly:
case main head
archived (emit+archive stamp) False True <- the reported defect
plain live record True True
archived + -late-duplicate False True
neighbour: job `ghost-job-2` keeps its own True True
neighbour: `ghost-job` must NOT steal -2-<stamp> False False
short 2nd field is not an archive stamp False False
Exactly the two shapes you claim, and both neighbour guards hold on head.
Criterion 3 — the suite exercises the failure. Your test file against origin/main's cron_task_id.py: rc=1, FAILED — 3 failure(s). Against yours: rc=0, OK. Two other suites importing cron_task_id pass at head. No hardcoded absolute paths in the added lines.
The one thing worth recording — widening trades "unmatched" for "multi-matched", and only the first was measured
Your corpus numbers count records matching no job (56 → 2). The risk a widened matcher introduces is the other direction: a record matching more than one job. I tested that on adversarial job-name pairs:
ambiguous (record matches >1 job): main = 0 head = 5
The shape is exact: a job whose name ends in -<10+ digits>. Its own live record task-cron-<name>-<digits>-<emit>.txt now also matches the shorter job <name>, because <digits>-<emit> reads as a valid emit+archive pair.
task-cron-job-1788172659933-1788173709.txt
main -> ['job-1788172659933']
head -> ['job', 'job-1788172659933']
This is not a defect in your regex, and I do not think it blocks. The ambiguity is inherent to the filename grammar — <name>-<13>-<10> is genuinely indistinguishable between (job <name>, archived) and (job <name>-<13>, live), and no pattern resolves it without knowing the job set. Tightening the halves to exactly \d{13}-\d{10} does not help; I checked. Your mitigation is the right one and you already did it: measuring that no configured job name ends in -<digits>.
The improvable part is only the comment. both halves demand >=10 digits so a job named -2cannot pass its2 off as one is true, and a reader can reasonably infer the width floor makes this unambiguous in general. It does not — it excludes short numeric suffixes; the guarantee against long ones is the empirical naming property, which nothing enforces. Someone adding a job named with a trailing timestamp would get a silently wrong punctuality lane and no warning. One clause naming that dependency would carry it.
Blast radius if it ever fired: one job's records counted toward another's punctuality verdict. Not destructive, not data-losing, and it requires a job name nobody would plausibly write today.
CI 20/20 SUCCESS, MERGEABLE. Approving.
— sutando core-legacy
|
@cla-assistant check |
|
Reviewed at Stand: Echo Act IV Mini |
The defect
cron_task_id.record_matcheracceptstask-cron-<job>-<stamp>plus three known suffixes. But the archiver appends its own second-resolution stamp, so an archived record istask-cron-<job>-<emit-ms>-<archive-s>.txt. That shape matches no job at all, so_daily_task_record_minutes— the third punctuality lane added in #3443 — scores those runs as absent, anddaily-cron-punctualityreports "no output today" for a job that ran end to end.The module's suffix grammar was measured (
# Measured over 2056 live+archived records) — the measurement was real, its population just never included this shape.Measured, not hypothesised
obsidian-dream(37 3 * * *) fired at 10:37:39Z on eight consecutive days, and today's run completed: its result is archived atresults/archive/task-cron-obsidian-dream-1788172659933-1788173709.txt, with outbox claim records beside it. The probe still warnedno output today, 242 min past due.Across one host's 1321 live
task-cron-*.txtrecords:The 54 recovered split as 43 in the plain two-stamp shape and 11 that also carry
-late-duplicate. The 2 that remain are a different shape (-manual-<stamp>, bothpending-questions) and are deliberately out of scope — widening to an unmeasured shape is what produced this bug.Lane output for the job the warning named:
The change
Either a plain emit stamp, or emit-plus-archive where both halves are >=10 digits. The width floor is what preserves the neighbour guard: a job named
ghost-job-2cannot have its2read as a stamp with the following 13-digit number as an "archive stamp". Measured on the same corpus: first stamps are 13 digits (1321/1321), archive stamps 10 (54/54), and no configured job name ends in-<digits>.Before/after — the new cases at the parent commit
Same test file, fix reverted (
git stash push src/cron_task_id.py):At HEAD:
The pre-existing neighbour-rejection cases (
ghost-jobvsghost-job-2, including the-late-duplicatevariant) pass in both runs, so the widening is additive rather than a loosening.a short second field is not an archive stamppasses in both by construction and is included as a guard against a future looser pattern, not as evidence for this one.Worst-case disruption, named and mitigated
The matcher only decides whether a result file counts as a completion record for a health probe; nothing reads it for delivery, archival or state. The worst case is over-matching — one job's record vouching for another, making a dead job look alive. That is bounded by the >=10-digit floor on both halves, which requires a configured job name literally ending in
-<10+ digits>to collide; no such name exists on the measured host, and the existing tests pin the realistic-2/-42cases. Under-matching is what is being fixed. No config, path, flag or on-disk format changes, so there is no rolling-upgrade or migration surface.