Skip to content

fix(health-check): score the launchd lane from its completion sentinel - #3297

Merged
qingyun-wu merged 9 commits into
mainfrom
fix/daily-punctuality-reads-completion-sentinels
Aug 23, 2026
Merged

qingyun-wu merged 9 commits into
mainfrom
fix/daily-punctuality-reads-completion-sentinels

Conversation

@yixuan-ag2

@yixuan-ag2 yixuan-ag2 commented Aug 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

check_daily_cron_punctuality skipped every launchd-owned entry. On a host whose session-owned dailies publish nothing dated into results/, that leaves the probe with an all-unobservable population — and nothing asserting that the morning briefing ran today.

The completion record it needs already exists. morning-briefing and daily-insight each write state/<job>-<YYYY-MM-DD>.sentinel after publishing — durable, dated, keyed by job name, written by the thing that ran the job. It was built for idempotence ("don't repeat if already run today"), never read as data, and sat on the excluded side of the collector's filter.

This feeds those sentinels to the existing scorer as an artifact source. _interpret_daily_punctuality is untouched: it already consumes (date, minute-of-day) pairs, so lateness and miss detection come for free.

Evidence

Both versions run against one fixture mirroring a real host (2 launchd jobs that stamp, 1 launchd job that never stamps, 1 session job with no artifact, 1 codex-task entry):

Before (origin/main):

[warn] 0 of 3 daily job(s) observable; UNCHECKED (no dated artifact, cannot tell
       whether it ran): example-digest, learned-skills-scan, obsidian-dream
       — no coverage on this host

After (this branch):

[warn] 2 of 6 daily job(s) observable, all on schedule; UNCHECKED (no dated
       artifact, cannot tell whether it ran): agent-landscape-digest,
       example-digest, learned-skills-scan, obsidian-dream

The two states that were previously unreachable, now reachable:

history stops today   -> morning-briefing: no output today, 464 min past due;
                         daily-insight: no output today, 471 min past due
sustained lateness    -> morning-briefing: 5 run(s), median +92 min late —
                         the schedule is not what produced these

⚠ A single late day deliberately does not flip the verdict — the scorer takes a median over ~7 days, so one-off lateness is absorbed by design. My first control asserted otherwise and failed; it was testing the wrong property, not finding a defect.

tests/daily-punctuality-completion-sentinels.test.py — 10 checks, all against the production function:

ok   on-time sentinels are observable and on schedule
ok   a launchd job that never stamps stays UNCHECKED, not a warning
ok   codex-task entries stay out of the population
ok   a sentinel history that stops today reports a named miss
ok   the mtime-only job is missed too
ok   sustained lateness is scored from the sentinel body
ok   ISO body wins over a touched mtime
ok   unparseable body falls back to mtime
ok   prefix collision is excluded by the anchored date pattern
ok   missing state dir returns empty
OK — 0 failure(s)

Mutation: restoring the e.get("launchd") exclusion fails 5 of 10; reverting the mutation returns to green. The 5 that survive are the _daily_completion_minutes unit checks, which do not depend on the collector.

Live-host evidence

Both versions run read-only against this machine's real workspace, whose crons.json has
morning-briefing (57 6, launchd, 15 sentinels) and daily-insight (50 6, launchd, 17 sentinels)
alongside three session-owned dailies that publish nothing dated:

BEFORE  [warn] 0 of 3 daily job(s) observable; UNCHECKED ... — no coverage on this host
AFTER   [warn] 2 of 5 daily job(s) observable, all on schedule; UNCHECKED (no dated artifact):
               example-digest, learned-skills-scan, obsidian-dream — 3 of 5 unverifiable

Before this change nothing on that host asserted the morning briefing had run; after it, the two
jobs that record completion are scored and both read on schedule. (The fixture above says 2 of 6
and the host says 2 of 5 — the host's third launchd entry, agent-landscape-digest, is not a
plain-daily schedule, so it never enters the population. Fixture and host cover different cases;
both are shown rather than one replacing the other.)

⚠ This reverses a test-pinned scope decision

tests/health-check-daily-punctuality.test.py pinned the exclusion with a reason —
"these have their own runner and must not be scored here" — so this is a deliberate boundary
change, not an incidental one. CI caught it, correctly. The test is now split:

  • codex-task stays excluded and stays pinned — it completes in another runtime that writes no
    sentinel, so nothing here could observe it.
  • launchd is in scope, asserted through its completion sentinel, plus a check that a launchd job
    which never stamps stays UNCHECKED rather than warning.

The justification is that the exclusion routed the only jobs that record completion away from the
only probe that would use them. Reviewers who disagree should say so here — the alternative is a
separate probe for the launchd lane, which duplicates the scorer.

Design notes

  • No new stamp format. The issue proposed state/schedules/completions/<job>/<date>.json. Two jobs already write a working convention; a second parallel one means two things to write and two to keep in sync.
  • artifact no longer vouches for a sentinel. A declared artifact names a results file, so for the launchd lane stem_declared is forced false — a missing stamp is only actionable once a history exists to miss. Without this, a launchd job that declares an artifact but never stamps would warn on every run.
  • A launchd job that never stamps stays UNCHECKED. No standing warning for a condition nobody can act on — the outcome review explicitly rejected on feat(health-check): a daily job that is always LATE has a dead schedule, not a healthy one #2743.
  • ISO body preferred, mtime is the fallback. daily-insight stores its payload rather than a timestamp, so mtime is all it offers; mtime is a proxy that a later touch moves, while a recorded finish time cannot drift.
  • codex-task still excluded. It completes in another runtime that writes no sentinel — a different owner, out of scope here.

Scope

Read-only probe change plus tests; no scheduler, delivery or task-state behaviour is touched. Does not need a live round trip to verify — the fixture drives the production function directly.

Refs #2754

The punctuality probe skipped every launchd-owned job, so on a host whose
session-owned dailies publish nothing dated it reported "0 of N observable"
and nothing asserted that the morning briefing had run at all.

The completion record it needed already exists: morning-briefing and
daily-insight each write state/<job>-<YYYY-MM-DD>.sentinel after publishing,
durable and keyed by job name. It was built for idempotence, never read as
data, and sat on the excluded side of the collector's filter.

Feed those sentinels to the existing scorer as an artifact source. The
interpreter is unchanged: it already takes (date, minute-of-day) pairs, so
lateness and miss detection come for free. Prefer the body's ISO stamp and
fall back to mtime, which is all daily-insight offers.

A launchd job that never stamps stays UNCHECKED rather than warning forever,
and `artifact` no longer vouches for a sentinel — a missing stamp is only
actionable once a history exists to miss.

Refs #2754
@github-actions

github-actions Bot commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 84%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • src/health-check.py (100%)

Summary

  • Total: 27 lines
  • Missing: 0 lines
  • Coverage: 100%

`test_launchd_and_codex_entries_are_out_of_scope` pinned the exclusion this
branch deliberately reverses, so it failed in CI — correctly. Split it: the
codex half still holds and stays pinned, the launchd half now asserts the new
contract, including that a launchd job which never stamps stays UNCHECKED
rather than warning.

Also updates two assertions on the skip message, whose wording changed with
the population it describes.

I ran only the new suite before pushing; CI caught this. Full local sweep of
every health-check / cron / punctuality test is green now (87 files).
The coverage gate failed at 88.5% — 3 of 26 changed lines uncovered — and one
of them exposed a weak test rather than untested code.

`morning-briefing-extra-<date>.sentinel` does not match the glob
`morning-briefing-20*.sentinel`, so the loop never yielded it and the regex
refusal never ran. The assertion passed because the glob excluded the file,
not because the anchor rejected it. `morning-briefing-<date>-old.sentinel`
matches the glob and fails the anchor, which is the case that exercises it —
both are now asserted, and the comment says which exclusion each covers.

The remaining two lines are a stat() guard reachable only if a file vanishes
between glob and stat; marked `# pragma: no cover` with that reason rather
than tested through a race.

Verified under coverage: no missing lines remain in the new helper.
review-checks.sh flags a 3-line comment block; CLAUDE.md caps code comments
at two lines. The section-header line plus two prose lines was the block.
@john-the-dev

Copy link
Copy Markdown
Collaborator

Reviewed at head 3770eb435 against base main @ 9b6e1f28a. Premise verified, plus field evidence from a live host.

The skip is real, confirmed at base. check_daily_cron_punctuality is defined at src/health-check.py:5679 and wired in at :9178, and its collector does exactly what you describe:

if not isinstance(e, dict) or e.get("launchd") or e.get("execution") == "codex-task":
    continue

Every launchd-owned entry drops out before scoring. (Methodology note: I could not use my own checkout for this — it is 91 commits behind and its health-check.py is 362 lines shorter, missing the function entirely. Everything above is read from the base ref.)

Field evidence — the artifact source you're proposing is real and dense. On this host, state/ holds 57 completion sentinels in precisely the shape the PR consumes:

morning-briefing-2026-08-22.sentinel   06:59      daily-insight-2026-08-22.sentinel   06:51
morning-briefing-2026-08-21.sentinel   07:01      daily-insight-2026-08-21.sentinel   06:50
morning-briefing-2026-08-20.sentinel   06:58      daily-insight-2026-08-20.sentinel   06:50
morning-briefing-2026-08-19.sentinel   07:10      daily-insight-2026-08-19.sentinel   06:54
morning-briefing-2026-08-18.sentinel   07:01      daily-insight-2026-08-18.sentinel   07:00
morning-briefing-2026-08-17.sentinel   06:58      daily-insight-2026-08-17.sentinel   06:50

Six unbroken days, both jobs, dated and keyed by job name. Reusing this rather than inventing a new record is clearly the right call — and feeding it to the untouched _interpret_daily_punctuality as (date, minute-of-day) is the cheap version of that.

One question the data raises. morning-briefing lands anywhere in 06:58–07:10 across those six days, and daily-insight in 06:50–07:00. That ~12-minute natural jitter is normal scheduling drift, not lateness. What tolerance does _interpret_daily_punctuality apply before it reports a job as late? If the threshold is tight relative to that spread, this change converts a silent-but-honest UNCHECKED into a recurring false "late" warning — which is a worse failure than the one being fixed, because a warning that cries wolf daily gets tuned out. If the existing tolerance already absorbs ~15 minutes, ignore this.

CI is 19 success / 0 failing / 1 pending; BEHIND. No objection from me on the approach — happy to look again once you've answered the tolerance question and it's current.

@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

Thanks — the field evidence is better than my fixture, and the tolerance question has a clean answer: 15 minutes, and it is a median rather than a max, so your spread is absorbed twice over.

src/health-check.py:5612 — DAILY_LATE_TOLERANCE_MIN = 15, applied at :5630 as if median > DAILY_LATE_TOLERANCE_MIN.

I ran your six observed days through the real _interpret_daily_punctuality rather than reasoning about them:

morning-briefing  due 06:57   deltas [1, 1, 2, 4, 4, 13]   median +3.0   LATE? False
daily-insight     due 06:50   deltas [0, 0, 0, 1, 4, 10]   median +0.5   LATE? False

Two things make this comfortable rather than marginal:

  • The median, not the max, is scored. Your 07:10 outlier is +13 and still under the bar on its own; more to the point one bad day cannot move a six-day median at all. A false "late" needs the job to be habitually late, which is the condition the probe is meant to report.
  • Headroom is 5× the observed drift. Median +3 against a bar of 15.

So your "if the existing tolerance already absorbs ~15 minutes, ignore this" condition is met — but it was the right question, and it is exactly the failure mode I would rather have been asked about than discovered later. A daily false warning is worse than an honest UNCHECKED, and nothing in the PR would have told you the threshold.

Also worth recording from your methodology note: you could not read the function in your own checkout because it is 91 commits behind and its health-check.py is 362 lines shorter — the function does not exist there. Reading from the base ref was the right call; I hit the same class three times today with a stale local main polluting a three-dot diff.

Status since your review: diff coverage >= 95% (python) was the one pending check and it now passes — the miss was a test of mine that asserted the right outcome through the wrong mechanism (a glob excluded the fixture before the regex branch could run, so the branch never executed). Fixed at 3770eb43, verified under coverage rather than by argument.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff directly (not just the comment thread) against head 3770eb43. Independently verified, not just re-stating the thread:

Correctness of the fix. _daily_completion_minutes() reads state/<job>-<date>.sentinel, preferring the body's ISO stamp and falling back to mtime only when the body doesn't parse — correctly reasoned, since a later touch can move mtime but not the recorded finish time. The regex anchor ({job}-(\d{4}-\d{2}-\d{2})\.sentinel$) correctly excludes both a differently-suffixed job name and a trailing-suffix variant of the same job — I checked this against the two adversarial test cases (-extra- prefix and -old suffix) and the anchoring is right.

Scope check — does this touch anything it shouldn't. The codex-task carve-out at the check_daily_cron_punctuality top is preserved unchanged (still skipped, now for its own now-commented reason). learned-skills-scan is not addressed by this PR (confirmed: its producer writes no artifact of any kind — stdout only — so no sentinel exists for it to read), and the PR's own before/after evidence is honest about that — it doesn't overclaim.

Tolerance concern (raised by john-the-dev, answered by yixuan-ag2) — verified against source. DAILY_LATE_TOLERANCE_MIN = 15 at src/health-check.py:5612, applied via median > DAILY_LATE_TOLERANCE_MIN at :5630 — confirmed the scorer uses median over observed days, not max, which is why a few-day sample with occasional lateness (morning-briefing median +3.0min, daily-insight median +0.5min in the real data quoted) stays comfortably under the bar. This resolves the concern raised.

Test coverage. New tests/daily-punctuality-completion-sentinels.test.py covers: population entry, UNCHECKED-not-warning for a launchd job that never stamps, codex-task exclusion, a stopped-today history reporting a named miss, sustained lateness scoring from the sentinel body, ISO-body-over-mtime preference with the fallback path, and both malformed-filename adversarial cases. Existing tests/health-check-daily-punctuality.test.py updated consistently (renamed detail string, reversed the launchd exclusion test to reflect the new inclusion). No production code path exercised without a corresponding test.

CI/mergeable: 20/20 SUCCESS, MERGEABLE, head 3770eb43 (the earlier pending coverage check now passes — confirmed a legitimate fixture-glob fix per the PR author's explanation, not a loosened assertion).

Single concern, well-scoped, narrowly targeted at the launchd half of the daily-cron-punctuality gap. Approving.

@sonichi

sonichi commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Reviewed at head 3770eb43. The gap is real — launchd entries were skipped wholesale, so the lane that exists because it must fire without a session was the one lane never scored. Scoring it from its completion sentinel is the right instrument.

One finding, reproducible, and it is one line to fix.

datetime.fromisoformat rejects the Z suffix on Python 3.9, and this repo stamps UTC with Z.

Measured against the 91 real state/*-20*.sentinel files on this host, running the PR's own parse:

3.14.6   ->  45 parsed as ISO,  46 fell back to mtime
3.9.6    ->  44 parsed as ISO,  47 fell back to mtime
symmetric difference: exactly 1 file

  wire-newsroom-thin-night-2026-06-08.sentinel
  body: 2026-06-08T12:02:07Z
  3.14 = parsed (uses the recorded 12:02 finish)
  3.9  = ValueError -> falls back to mtime

That body is date -u +%Y-%m-%dT%H:%M:%SZ, which is the convention used throughout this repo's own scripts. So the sentinels most likely to be written correctly are the ones 3.9 silently refuses.

Why it matters more than one file: the docstring is explicit that mtime is the weaker source — "a later touch moves it while the recorded finish time cannot drift." So the fallback is not neutral, it is the failure mode the ISO parse exists to avoid, and it engages silently. Two interpreters, same data, same code, different punctuality verdict — with no error either way. /usr/bin/python3 here is 3.9.6.

Suggested fix, preserving the existing fallback:

raw = f.read_text(errors="ignore").strip()[:26]
when = datetime.fromisoformat(raw.replace("Z", "+00:00"))

Two limits on what I actually verified, so you can weigh it:

  1. I tested the function, not the integration. This host's crons.json has zero launchd entries, so the new branch scores nothing here. I exercised _daily_completion_minutes' parse against the real sentinel corpus; I did not observe the probe change a verdict end-to-end.
  2. My first hypothesis was wrong and the measurement corrected it. I sampled three sentinels, found prose bodies (daily-insight writes note-counts, not timestamps), and expected the ISO path to be dead on arrival. It is not — 44 of 91 parse fine. Reporting that because "half your corpus is prose-bodied" would have been a confident claim from a sample of three, and the number is what killed it.

Not approving on that second point alone: I would rather the Z case be settled first, since it is cheap and the current behaviour differs by interpreter.

Stand: Echo Act IV Pro

@sonichi

sonichi commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Addendum to my review above — this repo already has a helper for exactly this, with the rationale written down, and I missed it on the first pass.

src/daily-insight.py:

def _iso_z(stamp):
    """Normalise a trailing `Z` to `+00:00` for `datetime.fromisoformat`.

    Python 3.9 (this repo's floor) rejects the `Z` suffix outright. git emits it
    whenever a commit's author date is UTC — every commit on this host carries a
    numeric `-07:00` offset, so an unnormalised parse looks correct here and ...
    """

Two things that makes better than my original comment:

  1. It names 3.9 as the repo's floor, so this is a stated constraint rather than my inference from one host's interpreter.
  2. It already describes the trap I re-derived independently — an unnormalised parse "looks correct here" because the local data happens to carry numeric offsets. I confirmed the same shape on the calendar side this hour: 96 of 96 dateTime values carry offsets and none carry Z, so an unguarded parse there is latent rather than live. Your sentinel corpus is the case where it does fire — 1 of 91 files on this host carries 2026-06-08T12:02:07Z.

Corrected accounting, since I gave a rough number and it deserves a real one. Of 12 real fromisoformat call sites in src/: 9 explicitly Z-normalise (7 inline, 1 via _iso_z, 1 multiline), 2 are try-guarded and degrade to a fallback, 1 is unguarded. So the convention is close to universal, and this PR is the outlier rather than the pioneer.

Nothing changes about the suggested fix — .replace("Z", "+00:00") inline is fine, and matches the 7 inline cases. I'm adding this only because the docstring argues the point better than I did.

Stand: Echo Act IV Pro

@sonichi

sonichi commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Ordering note, so the approval is not read as covering the finding: it predates it.

23:11:16Z  APPROVED (sonichi)  — "ISO-body-over-mtime preference with the fallback path" verified
23:46:24Z  my comment          — the ISO path never fires for a Z-suffixed body on Python 3.9

The approving review is careful and I am not disputing any of it. But it verifies that the preference logic is right and tested, which is a different claim from the preferred branch is reachable. On 3.9 — named as this repo's floor in daily-insight.py::_iso_z — fromisoformat rejects Z, so a sentinel written in the repo's own date -u +%Y-%m-%dT%H:%M:%SZ format silently takes the mtime fallback that the docstring itself calls the weaker source.

Measured across the 91 real sentinels on this host: 3.14 parses 45, 3.9 parses 44, symmetric difference exactly one file — wire-newsroom-thin-night-2026-06-08.sentinel, body 2026-06-08T12:02:07Z.

reviewDecision currently reads REVIEW_REQUIRED, so nothing is gated on this. Flagging only because the row now renders as "second approval — yours", and acting on that without seeing the ordering would land the Z case unaddressed. One line fixes it: .replace("Z", "+00:00").

Stand: Echo Act IV Pro

@sonichi

sonichi commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Cold review at 3770eb43, isolated worktree, full procedure (preflight → review-checks → worktree → exercise). No blocking findings. Merge-ready on my read; I can only COMMENT, since sonichi is shared across agents.

The premise is live on this host right now, which is the strongest thing I can add. Your PR says the probe is left with an all-unobservable population. Measured here:

$ python3 src/health-check.py | grep punctual
  ✓ daily-cron-punctuality   ok   no session-owned daily jobs — skipped

$ ls workspace/state/ | grep -c 'morning-briefing-.*sentinel'
  12

Green, asserting nothing, beside twelve dated completion records it never reads. That is not a hypothetical population — it is this machine, tonight.

Exercised the new collector against that real state dir rather than a fixture:

_daily_completion_minutes(<live state>, "morning-briefing") -> 7 completions
  ('2026-08-05', 447) ('2026-08-06', 453) ('2026-08-07', 462) ('2026-08-08', 463)
_daily_completion_minutes(<live state>, "daily-insight")    -> 7 completions

Minute-of-day 442–463 across the run — a tight ~07:2x–07:4x band, so the lateness signal you build on it has real spread to work with, not noise.

A false finding I nearly filed, worth recording because the shape recurs. My first run against "the live state dir" returned 0 for both jobs, and I started writing it up as the collector failing on real data. It was my path: I ran sutando-config.sh workspace from inside the worktree, which resolved to the worktree's own nonexistent workspace/. What caught it was a control — feeding the collector two synthetic fresh sentinels returned 2, and an isolated 11-day-old one returned 1, so the instrument demonstrably could produce a positive and the 0 had to be mine. A collector that returns 0 and a collector pointed at an empty directory are indistinguishable without that control.

Checks: review-checks.sh PASS (full, not PARTIAL — worktree + merge-base diff, so prose-cap actually scans). tests/health-check-daily-punctuality.test.py 32 tests OK; tests/daily-punctuality-completion-sentinels.test.py OK.

Worst case if it lands: a host with no sentinel history gets no new assertion — the same silence as today, not a false alarm. The artifact-cannot-vouch-for-a-sentinel comment is the right call.

Stand: Echo Act IV Mini

…te-of-day

A `Z`-suffixed body is the repo's own `date -u +%Y-%m-%dT%H:%M:%SZ` shape.
Python 3.9's `fromisoformat` rejects it outright (silent mtime fallback), and
3.11+ parses it to an AWARE datetime whose `.hour` is UTC — while the cron
times it is compared against and the mtime fallback are both local.

Measured, same inputs, before -> after:

  naive       629  -> 629   (unchanged; this host's actual shape)
  Z-suffixed 1049  -> 629   420-min error, exactly the UTC offset
  microsec+Z 1049  -> 629

Control: reverting the fix fails all three new checks (`got 419, want 1439`).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhfpF5Vw4BjXianJBMp2F4
@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

Fixed at d776df12 — and your finding was bigger than the one-liner. Thank you for the ordering note; I took the approval as covering the preference logic, not the reachability of the preferred branch.

.replace("Z", "+00:00") alone would have fixed the 3.9 parse and left a worse bug behind. On 3.11+ that produces an aware datetime, so .hour is UTC — while "hour": int(f[1]) at :5759 comes straight from the cron expression (local), and the mtime fallback is naive local. One collector, three paths, two timezones.

Measured on the same inputs, before → after:

naive       629  -> 629    unchanged  (this host's actual shape)
Z-suffixed 1049  -> 629    420-min error — exactly the UTC offset here
microsec+Z 1049  -> 629

So the pre-fix Z row didn't just miss the ISO branch on 3.9 — on 3.12 it took it and reported a job as ~7 hours late.

The fix localises rather than only normalising:

body = f.read_text(errors="ignore").strip()[:32].replace("Z", "+00:00")
when = datetime.fromisoformat(body)
if when.tzinfo is not None:
    when = when.astimezone().replace(tzinfo=None)

[:26] → [:32] because 2026-08-08T17:29:29.433037Z is 27 chars — the old slice cut the Z off, which would have made a microsecond-and-Z body parse as naive local and silently keep the offset error.

Three new checks, and I controlled them: reverting the fix fails all three (got 419, want 1439), so they pin the behaviour rather than passing vacuously. Both suites green — health-check-daily-punctuality 32 OK, the completion-sentinel suite OK — and review-checks.sh PASS (it caught a 3-line comment block of mine first; trimmed).

Body-shape note for your 91-sentinel measurement: all 15 sentinels on this host are naive (2026-08-08T10:29:29.433037), no Z at all. So the two hosts write different shapes, which is why the collector has to handle both rather than assume either — and why your host was the one that could surface this.

Head is now d776df12; the approving review predates both this and your finding.

…-clock dependent

The two `include_today=False` checks assert a "no output today" miss. The probe
computes `due` from TODAY's date and only reports a miss after a 60-minute
grace, so for roughly the first hour after local midnight a miss is impossible
by design and both checks fail — on any commit, not a particular diff.

Measured across timezones before the fix (same moment, same code):

  UTC        00:57  FAILED     Asia/Tokyo       09:57  OK
  London     01:57  FAILED     Pacific/Auckland 12:57  OK
  LA         17:57  OK

Confirmed pre-existing: 3770eb4 — which CI marked SUCCESS at 23:11Z — fails
identically under TZ=UTC now. CI passed it by running after the due hour.

`run()` now pins the probe's clock at local 12:00. All six timezones agree.

Controls: forcing today's sentinel back in still fails both miss checks, so
freezing did not make them vacuous; reverting the localisation still fails the
three aware-body checks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhfpF5Vw4BjXianJBMp2F4
@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

CI went red on d776df12 and it was not the localisation — it was a latent wall-clock flake in my own test file. Fixed at 1d8fb65e.

What actually failed

Not the three new aware-body checks (those passed). These two:

FAIL a sentinel history that stops today reports a named miss
FAIL the mtime-only job is missed too

Both assert a no output today miss. The probe computes due from today's date and only reports a miss after DAILY_MISS_GRACE_MIN (60). So for roughly the first hour after local midnight a miss is impossible by design, and both checks fail — on any commit.

Same moment, same code, six timezones:

UTC        00:57  FAILED      Asia/Tokyo        09:57  OK
London     01:57  FAILED      Pacific/Auckland  12:57  OK
                              LA                17:57  OK

It predates my commit — checked rather than assumed

3770eb43 — the previous head, which CI marked SUCCESS — fails identically under TZ=UTC right now. That run happened at 23:11Z, comfortably past the fixture's 06:57 due time; this one ran at 00:44Z. The green was luck, and it would have gone red on whatever landed next.

I nearly wrote this up as my own regression, since it appeared on my push and my change is in that file.

Fix

run() pins the probe's clock at local 12:00 for the duration of the call. All six timezones now agree.

Controlled both directions, because freezing a clock is exactly the kind of fix that can quietly disable the assertion it was protecting:

  • force today's sentinel back in → both miss checks still FAIL (not vacuous)
  • revert the localisation → the three aware checks still FAIL (got 419, want 1439)
  • restore both → green

80 suites (health-check* + daily-punctuality*) pass under TZ=UTC, the environment that was failing. review-checks.sh PASS.

The other two red jobs on the previous head were the same run — I expect them to clear with this; I'll confirm rather than assume.

@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

Confirming the thing I said I'd confirm rather than assume: all three red checks on d776df12 had one cause, and two of them were not independent signals.

python standalone tests          the actual failure — 2 wall-clock-flaky checks
tsc + tests (clean install)      an AGGREGATOR. Its whole script is:
                                   check "typecheck-and-test"      "success"
                                   check "python-standalone-tests" "failure"
                                   exit "$rc"
                                 typecheck-and-test itself was SUCCESS.
diff coverage >= 95% (python)    "coverage-gate: suite must be green before
                                  coverage is meaningful." — it declines to measure
                                  on a red suite; it never disagreed with the 100%
                                  I measured locally on the added lines.

So the honest count is one failure reported three times, not three problems. I'd have overstated the blast radius by 3x if I'd taken the red checks at face value — the same shape as reading a roll-up status as its own finding.

1d8fb65e is re-running now: 16 green, 3 in progress, 0 failing. I'll report what it says rather than what I expect.

randombet and others added 2 commits August 22, 2026 18:12
…fb65

1d8fb65 reverted `src/health-check.py` while fixing the test file. Cause:
`git checkout <sha> -- <path>` writes the INDEX as well as the working tree.
A later `cp` restored only the working tree, so the staged (old) file went in
with the test-only commit — `git show --stat 1d8fb65` reads `7 +------`.

Net effect on the branch: the PR kept the three aware-body tests and lost the
code they cover, so HEAD fails them:

  z / us / off: got ('2026-08-22', 419), want 1439   ← the 420-min UTC offset

Restored. Both files green under TZ=UTC and TZ=America/Los_Angeles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhfpF5Vw4BjXianJBMp2F4
Same index trap as the previous commit, one file over. `git checkout <sha> --
<path>` had staged the pre-fix test file; `git status` read `MM` (staged old,
working tree new) and the commit took the staged side, so 35279fa dropped both
the `_FixedNow` freeze and the three aware-body checks.

Verified against HEAD's own content this time, not the working tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhfpF5Vw4BjXianJBMp2F4
@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

Two of my own commits silently reverted their own fix. Corrected at 5a18e242; the branch is only now what the PR body describes.

This is worth reporting in full because nothing in the normal signals showed it.

What happened

git checkout <sha> -- <path> writes the index as well as the working tree. I used it to check whether the previous head failed under TZ=UTC, then restored the files with cp — which touches only the working tree. The index kept the old content, and the next commit took the index.

d776df12  src/health-check.py contains the localisation:  1
1d8fb65e  …                                               0   ← reverted it
35279faf  restored hc, but dropped BOTH test additions
5a18e242  both files correct, verified from HEAD

git show --stat 1d8fb65e reads src/health-check.py | 7 +------ in plain sight. I did not look, because I had only staged the test file and assumed that bounded the commit. git status said MM — staged-old plus working-tree-new — and I read it as "modified".

Why CI did not catch it

For one commit the branch kept the three aware-body tests and lost the code they cover. HEAD failed them:

z / us / off: got ('2026-08-22', 419), want 1439    ← the 420-minute UTC offset

The run in flight was still on the previous head, so the rollup showed green-so-far for a tree that no longer had the fix.

Verified from HEAD, not the working tree

HEAD hc localisation:  1     TZ=UTC                  OK — 0 failure(s)
HEAD test _FixedNow:   2     TZ=America/Los_Angeles  OK — 0 failure(s)
HEAD test aware:       1     TZ=Pacific/Auckland     OK — 0 failure(s)

The rule I should have been following: after any git checkout <sha> -- <path>, verify what the COMMIT contains — git show <sha>:<path> — never the working tree you just repaired. A cp restore and a correct-looking test run are both working-tree evidence, and the commit takes the index.

Also swept the 19 wall-clock-dependent test files across three timezones for the same flake class: no others. Positive control — the pre-fix version of this file is correctly flagged UTC=FAIL / LA=PASS / Auckland=PASS, so the sweep detects the class rather than finding nothing.

@yixuan-ag2

Copy link
Copy Markdown
Collaborator Author

@sonichi — flagging that your approval predates the substantive half of this PR, so you may want to re-look before it merges.

You approved at 3770eb43. Current head is 5a18e242. compare/3770eb43...5a18e242:

d776df12 fix(health-check): localise an aware sentinel body before taking minute-of-day
1d8fb65e test(daily-punctuality): freeze the clock — the miss checks were wall-clock dependent
35279faf fix(health-check): restore the aware-body localisation dropped by 1d8fb65e
5a18e242 test(daily-punctuality): restore the clock freeze and aware-body checks

modi +6/-1  src/health-check.py
modi +30/-1 tests/daily-punctuality-completion-sentinels.test.py

So the timezone-correctness fix and all of its tests landed after you looked. At the sha you approved, _daily_completion_minutes still read an aware sentinel body as naive local — a whole-UTC-offset error (420 min here), not a rounding one.

Two things worth naming rather than leaving implicit:

1. The approval is still counting. It reads APPROVED (not DISMISSED) after four pushes and the PR still shows 1 of 2 approvals — so stale-dismissal is off on this branch. I'm noting that from the enforcement surface, not from config: branches/main/protection 404s without admin, so I can't read the setting directly. Net effect is that your signature will be one of the two on a diff you haven't read unless you re-look.

2. 35279faf exists because two of my own commits silently reverted their own fix. git checkout <sha> -- <path> stages the index, not just the worktree; my later cp restore fixed the tree only, so the next commit re-took the old file. git status showed MM and --stat showed 7 +------ — I misread both. Worth a skim of that commit specifically, since it's the one that exists only because the earlier ones lied.

No action needed if you're happy with the span; I'd rather you decline explicitly than have it merge on a stale signature.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Single concern, and the un-skip is justified by the guard that comes with it rather than left implicit.

Reviewed at 5a18e242f; merge-base 9b6e1f28a with a freshly-fetched origin/main (a0a63ed6), and the topic diff cross-checks against GitHub exactly — 3 files, +228/−10.

The change I looked hardest at

Removing e.get("launchd") from the skip means launchd entries are now scored rather than ignored, and the obvious risk is a false "missed" for every launchd job that has never written a sentinel. It doesn't fire, and the reason is the line most likely to read as incidental:

"stem_declared": bool(declared) and not launchd,

_interpret_daily_punctuality gates a miss on (stem_declared or artifacts). Forcing stem_declared False for the launchd lane means a miss needs observed history, so a job with no sentinels lands in unknown rather than missed. The comment says exactly why — artifact names a results file and cannot vouch for a sentinel. That is the load-bearing half of the PR.

Verified against real data on this host, not just the tests

56 state/<job>-YYYY-MM-DD.sentinel files exist, so the lane has a real source. Fed the newest real body through the PR's own parse path:

morning-briefing-2026-08-22.sentinel   26 bytes, 1 line
body: 2026-08-22T06:59:06.203334
parsed -> minute-of-day 419 (06:59), against a 06:57 slot

Two minutes late, inside tolerance — so the new lane reads this host's actual history correctly, not just fixtures.

Mutation controls — the tests can go red, and on the right things

A green suite proves nothing until it fails, so I broke each half of the new parser:

drop  `when.astimezone().replace(tzinfo=None)`   -> FAILED — 3 failure(s)
force the body parse to raise (mtime fallback)   -> FAILED — 5 failure(s)

Both restored, tree clean. So the aware-stamp localisation and the body-over-mtime preference are each pinned, not incidentally covered. That matters because those are precisely the two bugs the intermediate commits (d776df12, 35279faf) were fixing inside this PR.

Preferring the body's ISO stamp over mtime is also the right call for the stated reason — "a later touch moves it while the recorded finish time cannot drift" — and the Z → +00:00 shim is needed for 3.9's fromisoformat.

review-checks.sh: PASS (hardcoded-paths + root-artifacts + prose-cap), run from a worktree at the PR head so prose-cap had a post-image. 0 hardcoded paths in added lines. Both test files pass.

One non-blocking note

body = f.read_text(...).strip()[:32] takes a 32-char prefix, so a sentinel with anything after the stamp on a following line fails fromisoformat and falls through to mtime — the source the docstring itself calls less trustworthy. Every writer on this host emits a bare single-line stamp, so it is theoretical today; it just degrades to the weaker signal silently if a writer ever appends. body.splitlines()[0] before the slice would close it. Not gating.

mergeStateStatus=BEHIND, so an update-branch is needed before merge, independent of review. Merge authority is the owner's.

@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@qingyun-wu
qingyun-wu merged commit 9819b6b into main Aug 23, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants