Skip to content

feat(pending-questions): hand each decision-card click to exactly one consumer - #5946

Merged
sonichi merged 10 commits into
mainfrom
feat/pq-card-clicks
Oct 27, 2026
Merged

sonichi merged 10 commits into
mainfrom
feat/pq-card-clicks

Conversation

@sonichi

@sonichi sonichi commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Stack

PR 1 of 2, split from #5942 (kept as a draft for reference). This PR covers the click side only, based on main. PR 2 (the ask side: creating cards and delivering the question) stacks on this one.

What changed, and why

A pending question's decision card is a HITL choice requirement:

  • runtime pending-questions
  • guard and subject.ask_id set to the ask id
  • subject.choices mapping each action id to its consequence

The bridge owner-checks a click, records it, and forwards it to the core as a task headed hitl_click: true. This PR hands each recorded click to one consumer at a time, under a 10-minute lease, with at most one re-offer. In this PR, cards come only from a test-only path; the ask side does not create them yet.

python3 src/pending_questions_reader.py clicks (or pq.py clicks) reaches the adapter's new clicks, in skills/pending-questions/scripts/pending_questions_clicks.py:

  1. One lock, no pid probe. Every pass runs under one exclusive lock beside the HITL store, state/hitl/pending-questions-clicks.lock. Whoever holds the lock is the only live claimant. A claim still marked claiming was left by a pass that died, and is retried with the same token.

  2. Token-bound close. The pass claims the click with a token, saved atomically on the card. It then closes the row as Answered in one guarded write that:

    • requires the owner's Status to be open;
    • writes a plain Answered@<host> in Closed, and the token in Recovery (click:<token>@<host>). A reader that predates tokens, such as the reviewed base after a rollback, reads the row as closed and ignores Recovery.

    The owner's terminal Status is checked first: Answered or Resolved makes the click stale, whatever any mark says. Otherwise, a refusal counts as this claim's own earlier close only when the row's Recovery token is the same. Any other refusal makes the click stale and cancels its card. That covers the owner's own Status and a close by code such as resolve, so a recovered claim cannot take credit for an unrelated resolve.

  3. Fail closed. Only the room can say the owner's Status is open, so a click never closes through a local close record. The click is deferred when there is no room store, when the room is unreachable, or when the row is not in the room yet. Deferred means no chose, the claim kept, and a retry on the next pass.

  4. Leased handoff. On a close, the claim is recorded handed with handed_at before the choice is returned.

    • The choice prints once, to that caller alone, as [<ask-id>] chose <action-id> '<label>' -> <consequence> … ack with --token <token>.
    • Later passes print "handed out earlier, awaiting ack" until the lease expires.
    • The lease (pending_questions_lease.py, shared with feat(pending-questions): ask-owner asks by decision card when --default or --option is given #5947's pointer claim). Invariant: a lease expires exactly when the time credited to it reaches its window (HANDED_LEASE_SEC, 600 s). Each pass credits clamp(now - observed_at, 0, LEASE_MAX_STEP_SEC=300) and then sets observed_at = now. Consequences:
      • A backward step credits 0, so a rollback never expires a lease or raises NEEDS ATTENTION.
      • A forward spike of any size credits at most 300 s. Once the clock is corrected, normal time resumes: no freeze.
      • No single pass expires a fresh lease, because the step is smaller than the window.
      • Trade-off, chosen on purpose: catching back up after a rollback is credited again, so a rollback can over-credit by at most one step (300 s). The alternative, no double credit, keeps observed_at at its high water and freezes the lease for the whole span of a corrected spike, which is unbounded.
      • Only deltas between passes count.
      • A handoff or a re-offer starts a new lease.
      • Every clicks and every clicks --due (the health check) credits and saves the lease atomically under the clicks lock, so a restart resumes from the saved credit.
    • Re-offer: an expired handed claim is re-offered as chose to the next pass, once, with a fresh token and a new lease. A consumer that died after the handoff therefore no longer strands the choice.
    • After the re-offer: a claim still unacked is listed as "NEEDS ATTENTION — handed out twice, never acknowledged", with its token. It is never offered again, and clicks --due surfaces it.
    • Contract: at-least-once, at most twice. The core acts before it acks, so a slow consumer or a clock jump can let a second consumer receive the same chose. SKILL.md, CLAUDE.md and AGENTS.md now carry the rule: before acting on a chose, re-check the current state (already merged, already resolved, already done) and skip what is.
  5. Token-bound ack. … ack <ask-id> --token <token> settles the card. An ack carrying a superseded token, from a consumer that acked after its lease was re-offered, is refused as stale and changes nothing.

  6. Wakeup.

    • src/health-check.py runs on every proactive-loop pass and from the launchd fallback. It warns decision-card-clicks when the read-only probe clicks --due reports a pass owed: a click never processed, a deferred or interrupted claim, or an expired handoff whose re-offer is unused.
    • The warn names … clicks, which the core handles as for a hitl_click task. So a deferred click or an expired lease is retried with no new task.
  7. Lease clock. The clock is sampled at each handoff and re-offer, inside the lock, so lock wait and room I/O never shorten a lease.

  8. No text beside a live card. The reminder recognises a decision-card record. asked_recently treats a live card as the question's ask, and remind drops live-card questions from any text, --force included. This PR is the direct parent of the ask side (feat(pending-questions): ask-owner asks by decision card when --default or --option is given #5947), so rolling that back cannot bring a text ask beside a live card.

  9. A retired card wins. Each claim save re-checks the card under the HITL store lock and is refused for a card a concurrent resolve has retired. The pass then hands nothing over and returns no token ("settled meanwhile — not a decision").

  10. resolve retires the card. It cancels the question's card whenever it closes the row or the room refuses the close.

Docs: CLAUDE.md and AGENTS.md (Pending decisions), and the skill's SKILL.md ("Decision-card clicks").

Evidence

All tests are in tests/pending-questions-card-clicks.test.py (50 tests, OK). The crash and concurrency tests use real processes (os._exit) against the HITL store and the room database, through the adapter's own serve over the fake room's on-disk state.

Each item below fails with its fix removed:

  • Two concurrent consumers: exactly one chose line appears across two processes. With replay-until-ack put back, it fails 2 != 1.
  • Crash after the handoff, before the print: inside the lease it shows "awaiting ack"; at the lease it is re-offered once with a fresh token; the re-offer starts a new lease. With the lease removed, the test fails.
  • A second lease expiry: no second re-offer; the claim shows NEEDS ATTENTION with its token and can still be acked. With the cap removed, the test fails.
  • A future-dated handed_at: treated as fresh, including when the stored deadline has passed. With the guard removed, the test fails.
  • Two consumers after the lease expired: exactly one re-offer. This fails without the lease.
  • An ack inside the lease prevents any re-offer.
  • A late ack after a re-offer is refused as stale and changes nothing; the new token's ack settles the card. This fails without the token check.
  • Recovered claim versus an unrelated resolve: the claim is persisted as claiming; a real process runs resolve(Answered) and dies while retiring the card. The retried click is stale and never chose. With any code close accepted as the claim's own, it fails.
  • Owner Status against a matching token mark: the owner set Resolved, the Closed mark carries this claim's own token, and clearing the stale mark fails. The click is stale, never chose. With the Status-first check removed, the test fails.
  • Wakeup through the health check: a real process dies after the handoff. Inside the lease there is no warn. Past the deadline the check warns, and the command it names prints chose … (re-offered. A click deferred in an outage is handed over the same way.
  • Base-reader compatibility: the base's effective_status reads a clicked row as Answered. With the token written into Closed, the test fails.
  • Lease starts at the handoff: the clock reads 100 before the lock and 701 at the handoff, so the lease is {701, 0}.
  • Lease invariant table: W−1, W, W+1; a repeated time; a forward jump; a year's spike then correction, with no freeze; a backward step; a 1-second rollback; rollback then catch-up, over-credit at most one step; renewal resets both fields.
  • Spike recovery: spikes from an hour to 10¹² s each recover within window/step + 1 passes of normal time.
  • Retirement race: production resolve() injected before the handoff save, and before the re-offer save. Nothing actionable, no unsaved token.
  • Lease in the production writer: a 1-second rollback after a late re-offer never raises attention; a stamp just under a window ahead; a far-future stamp; a real-process restart resumes from the persisted lease.
  • Production due_clicks() spike trace: (300, not due), then corrected (300, not due), then (600, due).
  • Parent reader: a live card record is never reminded as text, --force included. With the filter removed, the test fails.
  • Outage with a terminal owner Status already set: the click is deferred, the claim stays claiming, and no local close is written. Once the room is back, the click is stale. With a local fallback put back, the test fails.
  • An unreachable room, or a row not in the room yet: the click is deferred, then handed over once the room answers.

Also covered, as before:

  • death after the claim or after the close;
  • owner Answered or Resolved followed by a contradicting click is stale;
  • a long label, a long question and duplicate labels print the exact operation;
  • resolve retires the card;
  • the reader CLI (including --token required, and a stale ack exiting 1) and pq.py forwarding.

Tests run

  • This PR's suite: tests/pending-questions-card-clicks.test.py, 50 tests, OK. health-check-fix-down-bridges and health-check-recover-cron fail the same way on the unchanged parent on this host; they depend on host state.
  • Existing suites, all pass:
    • every other pending-questions-* suite, and tests/skills/pending-questions/pq.test.py
    • every hitl-* suite
    • agent-api-pending-questions, check-pending-questions-collapse and friction-pending-questions-collapse
    • hermetic-env-structural
  • Changed-line coverage: 99.0% (306 of 309 changed executable lines), measured locally with coverage.py. The three missed lines are the warn's registration inside run_all_checks; its function is covered.
  • Checks: scripts/review-checks.sh passes, and agents-md-sync.sh --check passes.

Feature/documentation consistency

docs/catalog.json has no canonical document for pending questions. The behaviour is documented in SKILL.md and CLAUDE.md, as listed above.

Live path

No live round trip has been run: no card is created on main until PR 2.

🤖 Generated with Claude Code

… consumer

The click side of decision cards for owner pending questions, split out of
#5942. A card is a HITL `choice` requirement with runtime pending-questions,
guard and subject.ask_id = the ask id, and subject.choices = {action id:
consequence}; the ask side does not create them yet.

`src/pending_questions_reader.py clicks` (and `pq.py clicks`) reaches the
adapter's new `clicks` (scripts/pending_questions_clicks.py):
- every pass runs under one exclusive lock beside the HITL store; a claim
  left `claiming` by a pass that died is re-offered (no pid is probed);
- the pass claims the click, closes the row as Answered in one guarded write
  that also requires the owner's Status to be open (RoomDbStore.close
  require_open), and records the claim `handed` before returning it; the
  choice prints once with its exact action id and consequence;
- later passes, by any consumer, report it as awaiting ack, never as a new
  choice; `ack <ask-id>` settles it, or marks it acknowledged while its close
  is retried;
- a row already closed by the owner's Status, or by code without this
  click's claim, makes the click stale and cancels its card;
- `resolve` retires the question's card.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 94%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • skills/pending-questions/scripts/pending_questions_ask.py (100%)
  • skills/pending-questions/scripts/pending_questions_clicks.py (100%)
  • skills/pending-questions/scripts/pending_questions_lease.py (100%)
  • skills/pending-questions/scripts/pending_questions_remind.py (100%)
  • skills/pending-questions/scripts/pending_questions_room_db.py (100%)
  • skills/pending-questions/scripts/pending_questions_store.py (100%)
  • skills/pending-questions/scripts/pq.py (100%)
  • src/health-check.py (92.3%): Missing lines 13039
  • src/pending_questions_reader.py (100%)

Summary

  • Total: 309 lines
  • Missing: 1 line
  • Coverage: 99%

src/health-check.py

Lines 13035-13043

  13035         checks.append(_wiring)
  13036 
  13037     _card_clicks = check_decision_card_clicks()
  13038     if _card_clicks:
! 13039         checks.append(_card_clicks)
  13040 
  13041     _mem_index = check_memory_index_integrity()
  13042     if _mem_index:
  13043         checks.append(_mem_index)

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 6bc14fa (agent review): COMMENT.

These all hold, and each fails its test when reverted:

  • the lock;
  • the single actionable handoff;
  • owner Status beating a late click;
  • the exact consequence.

20/20 and 40 neighbouring suites pass, and CI is green.

Blocking: a consumer that dies after the handoff strands the choice for good. If it dies before ack (pending_questions_clicks.py:88-91,110-112), the row is already Answered, later passes print "awaiting ack" (pending_questions_reader.py:233), and CLAUDE.md:171 says to act only on chose. Please add a bounded re-offer, such as a lease or expiry on handed, with a test.

…and clicks fail closed in an outage

- A handed choice carries handed_at; one not acked within HANDED_LEASE_SEC
  (10 min) is re-offered to the next pass with a fresh token, so a consumer
  that died after the handoff no longer strands it. Exclusive within the
  lease, at-least-once after it.
- `ack` carries the claim token (`--token`); an ack with a superseded token
  is refused as stale and changes nothing.
- The click's row close writes its claim token into the Closed mark
  (`Answered+<token>@<host>`); a refusal counts as this claim's own earlier
  close only when the mark carries the same token, so a recovered claim can
  no longer take credit for an unrelated resolve.
- Clicks never close through a local close record: with no room store, an
  unreachable room, or no row in it yet, the click is deferred (no `chose`,
  claim kept) until the room can verify the owner's Status is open.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at a7e6bd4 (agent review): COMMENT.

What's fixed. The stuck handoff: there's now a 10-minute lease, a re-offer with a fresh token, token-bound acks and Closed marks, and outages defer. All 8 reverts fail their tests, 26/26 pass, and CI is green.

Still open: the action itself can run twice. CLAUDE.md:171 says act, then ack. So a consumer slower than the lease lets a second consumer act too, and the refused ack comes after both have acted. A forward wall-clock jump (time.time() at pending_questions_clicks.py:94,104) does the same. Also, the re-offer repeats every 10 minutes, not once.

Two options:

  • Accept at-least-once: say so in the PR and the skill doc, and make the click's action idempotent.
  • Close it with an ack-to-claim or a renewable lease, using boot id plus monotonic time.

sonichi and others added 2 commits October 26, 2026 23:59
…nd the at-least-once rule

- A handed claim past its stored deadline (`lease_until`) is re-offered ONCE
  (`reoffers`, MAX_REOFFERS = 1). After that it is listed by `clicks` as
  "NEEDS ATTENTION — handed out twice, never acknowledged", with its token,
  and never offered again.
- The lease compares against the stored deadline, and a `handed_at` in the
  future (the clock went back) counts as fresh, not expired.
- The contract is at-least-once, at most twice: SKILL.md, CLAUDE.md and
  AGENTS.md say to re-check current state (already merged, already
  resolved, already done) before acting on a `chose`.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tored deadline

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact current head 689f359f0ac69f08fcb42c951fefae9603889251.

Verdict: CHANGES_REQUESTED — not ready to merge. The bounded single re-offer, future-clock guard, token-bound ack, and ordinary outage paths now pass. One owner-authority failure remains.

Blocking — a terminal owner Status can lose to the matching click token after a partial stale-mark cleanup failure. reconcile_pending() catches a failed stale-mark clear() and continues (store.py#L548-L554); _close_for_click() ignores those reconciliation errors, then its GuardFailed handler accepts a matching Closed token before considering terminal owner Status (room_db.py#L328-L339).

Production-store repro: create the claim-owned Answered+claimtok@host close, then have the owner set Status to Resolved, make only RoomDbStore.clear raise, and run clicks() again:

owner_status=Resolved
actionable=True stale=False token=claimtok
close_message=... closed earlier by this claim

The card was changed to handed, so the obsolete click would be emitted as chose even though owner Status is authoritative. Worst case: the core executes an action the owner explicitly superseded. Please check terminal owner Status before granting a matching token credit, or fail closed on the relevant reconciliation error, and add this cross-product regression. Current tests cover the matching-token recovery and terminal-status outage separately, not together.

Evidence at this head: card-clicks 28/28; room DB 51 passed, 1 optional pycrdt skip; review checks and hardcoded-path scan passed; all 37 check runs and CLA are green. Those green suites currently miss the reproduced interaction above.

…lick-token logic

A refused click close now checks the row's owner Status first: Answered or
Resolved makes the click stale whatever its Closed mark says. Before, a mark
carrying the claim's own token (left when the stale mark could not be
cleared) was taken as this claim's close and handed the choice over beside
the owner's own answer.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact current head 9794fd11c35bd3445625b82990807b3b4ee57a14 against main at 20f3042b821815ba9e3f191a8d69bedb6282bc56.

Verdict: CHANGES_REQUESTED — not ready to merge. The requested owner-Status ordering fix is correct: the new regression passes, and removing the two Status-first lines makes that exact case fail. Three whole-path issues remain.

  1. Blocking — deferred and leased claims have no production wakeup. Lease expiry and deferred retry are evaluated only when clicks() is called. The only added production trigger is the incoming hitl_click task; the core is instructed to finish that task with [no-send], and the watcher has a restart sweep, not a timed lease callback. The crash path is therefore:

    first pass: handed, then dies before print/action
    restart inside lease: "awaiting ack" (not actionable)
    task result: [no-send] -> task retired
    lease expiry: no caller remains -> never re-offered
    

    A transient room outage strands a deferred claim the same way. The regression at tests/pending-questions-card-clicks.test.py:189-206 manually invokes clicks() at a future timestamp, so it proves the state can retry, not that production will retry it. Add a durable delayed wakeup/requeue and test the real task lifecycle through expiry/recovery.

  2. Blocking — the new room mark is not readable by the reviewed base during rollback/rolling upgrade. New code writes Answered+<token>@<host>, while only the new parser strips the token. Exact-base/head probe:

    raw_closed=Answered+0123456789abcdef@test-host
    base 20f3042b effective_status=Open
    head 9794fd11 effective_status=Answered
    

    After rollback, a clicked question reappears open and can be reminded again; old code also cannot close it because Closed is already non-empty. Keep the token in a backward-ignored field or use a transition that remains legible to the old parser, with a base-reader compatibility test.

  3. Blocking — a newly handed ten-minute lease can already be expired. now is sampled before the blocking lock and reused after lock wait plus any earlier card/room work. Controlled timing probe:

    sampled now=100; lock acquired / handoff at=701
    stored lease_until=700
    immediate next pass: actionable=true, reoffered=true
    

    Sample the clock inside the lock at each handoff/re-offer so the advertised lease begins when that transition is persisted.

Positive evidence: the prior matching-token/terminal-Status bug is fixed (head: actionable=False, stale=True; fix-removed control: actionable=True, stale=False). Focused checks passed 163 tests with one optional pycrdt skip; review-checks.sh, git diff --check, and the hardcoded-path positive control behaved correctly; current CI and CLA are green.

…egible close mark, a lease that starts at the handoff, and no text beside a live card

- Wakeup: src/health-check.py (every proactive-loop pass and the launchd
  fallback) warns `decision-card-clicks` when the core reader's new read-only
  `clicks --due` probe says a pass is owed (a click never processed, a claim
  left `claiming` by an outage or a death, or a handoff past its deadline
  with its re-offer unused), naming `… clicks` to run.
- Rollback: a click close writes a plain `Answered@<host>` in Closed and its
  token in Recovery (`click:<token>@<host>`), which the reviewed base reads
  as closed and ignores; the token is read back from the row on a refusal.
- Lease: the clock is sampled at each handoff and re-offer inside the lock,
  so lock wait and room I/O no longer shorten a new lease.
- Card records are recognised by the reminder here, the direct parent of
  the ask side: asked_recently treats a live card as the ask, and remind
  drops live-card questions from any text, --force included, so rolling the
  ask side back cannot bring a text ask beside a live card.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@keweichen, replying to your review at 9794fd11: all three findings are fixed in place at 6c57b3e. Each new test fails with its fix removed.

1. No production wakeup.

  • Fix: the wakeup is wired into an existing periodic path, src/health-check.py. That runs on every proactive-loop pass and from the launchd fallback, so no new daemon is added. It warns decision-card-clicks whenever the core reader's new read-only probe clicks --due reports a pass owed: a click never processed, a claim left claiming (by an outage or a death), or a handoff past its deadline whose single re-offer is unused. The warn names python3 src/pending_questions_reader.py clicks, to be handled as a hitl_click task.

  • Tests: Wakeup runs your crash path end to end.

    1. A real process dies after the handoff.
    2. Inside the lease, the health check raises nothing.
    3. Past the deadline it warns, and running the command it names prints chose … (re-offered.
    4. A second check owes nothing.

    A click deferred in an outage is handed over the same way.

  • Limit: a host that never runs the health check retries only on the next click task. SKILL.md says so.

2. The mark isn't readable to the base after a rollback.

  • Fix: a click close now writes a plain Answered@<host> in Closed and its token in Recovery (click:<token>@<host>), in the same guarded write. The base reader sees Closed=Answered and ignores Recovery. A refused close reads the token back from the row; if that read fails, the click is deferred.
  • Test: RollbackAndClock.test_a_click_close_reads_closed_to_the_base_reader runs the base's reader logic (_mark / effective_status from 20f3042) on the row a click left, and gets Answered.

3. A new lease can already be expired.

  • Fix: the clock is now sampled at each handoff and re-offer, inside the lock, after the room I/O.
  • Test: your probe. The clock reads 100 before the lock and 701 at the handoff; lease_until is now 1301, and the next pass at 702 is not a re-offer.

This round also moves the card-record reader (card_sent / card_state / card_live) and the reminder's "never text beside a live card" filter into this PR. That fixes #5947's rollback finding at its direct parent. Tests: card-clicks 40/40, plus every pending-questions, hitl and proactive-loop budget suite. health-check-fix-down-bridges and health-check-recover-cron fail identically on the unchanged parent on this host; they depend on host state.

… cause

The new health check's name must be in recovery_issues.HEALTH_CAUSES, or its
warn leaves the host as an unknown cause; tests/recovery-issues.test.py pins
that inventory against the check names in src/health-check.py.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head afe5424861c8d0e60141a84dbfa7e0e43d40c198 against base 20f3042b821815ba9e3f191a8d69bedb6282bc56. APPROVE.

The prior blockers are resolved: due/expired leases now surface through the recurring decision-card-clicks health probe; terminal projection remains readable after rollback while token detail stays in Recovery; and lease time is sampled under the production click lock. The only delta since the previously reviewed head registers decision-card-clicks in the recovery inventory, which fixes the deterministic inventory failure.

Evidence at this head: five blocker-focused regressions pass; tests/recovery-issues.test.py is 12/12; tests/health-check-recovery-telemetry.test.py is 16/16; the relevant card-click suite was 40/40 at the patch-equivalent prior head; mutation checks caught inventory removal and mark-before-status ordering. Current CI and CLA are green (one intentional skip). The remaining model interpretation of a recurring WARN is disclosed and acceptable for this model-driven loop.

@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@qingyun-wu could you give this PR a second review at afe5424? keweichen approved it at that head; the branch rule needs one more approval. You reviewed earlier rounds. What changed since then: owner Status now wins before token recovery; a health-check wakeup retries deferred and expired leases; the Closed mark stays readable after rollback; and the lease deadline is set after the lock is taken. Each fix has a test that fails without it.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at afe5424 (agent review). The suite passes 40/40, each revert fails its test, and CI is green.

Verified:

  • one re-offer, then NEEDS ATTENTION;
  • health-check's decision-card-clicks wakeup;
  • owner Status is checked before the token logic;
  • the at-least-once rule is documented.

Still open:

  • pending_questions_clicks.py:98 keeps a future-dated handed_at fresh, so a clock that was ahead holds the click for the whole skew. I reproduced it: with the clock a year ahead, the click was still stuck at +30 days.
  • due_clicks (:157) skips that case and NEEDS ATTENTION, so health-check never surfaces either one.

Suggested fix: clamp the future stamp, and include both cases in due_clicks.

…from first sight; due_clicks surfaces it and NEEDS ATTENTION

A handed_at ahead of this clock is still never expired on sight, so a
backward step cannot steal a live lease; but the first pass that sees it
records skew_seen_at, and it expires one lease after that, so a clock that
ran far ahead no longer holds the click for the whole skew. A stored
deadline beyond one lease is clamped to handed_at + lease. due_clicks (and
so the health check's decision-card-clicks warn) now also surfaces an unseen
skewed claim and a NEEDS ATTENTION one.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@qingyun-wu, replying to your review at afe54248: both points are fixed at d9f49af, reconciled with keweichen's rule that a backward clock step must not steal a live lease.

1. A future-dated handed_at held the click for the whole skew.

  • Fix: a handed_at ahead of this clock is still never expired on sight. That keeps keweichen's no-steal rule. But the first pass that sees it records skew_seen_at, and the lease expires one lease after that. A stored deadline beyond one lease is also clamped to handed_at + lease.
  • Tests (in SkewedClock):
    • Clock a year ahead: not actionable when first seen; still held at first sight + 599 s; re-offered at first sight + 600 s. It fails with the fix removed.
    • Clock stepped back 1 s: no steal on sight, and the original lease still ends where it always did. With a naive clamp that expires future stamps on sight, it fails.
    • First sight recorded from the future: taken again at the current time.
    • Over-long stored deadline: clamped.

2. due_clicks skipped both cases.

  • Fix: it now includes an unseen skewed claim and a NEEDS ATTENTION claim (its single re-offer used and still unacked). The health check's decision-card-clicks warn therefore surfaces both. The warn now reads "owe a pass or need attention".
  • Tests: test_the_due_probe asserts both. It fails on the previous head.

Tests: card-clicks 44/44, and every pending-questions and hitl suite plus recovery-issues pass. #5947 is rebased onto this head and gets the same bound for a pointer claim stamped ahead of the clock, the nit at pending_questions_ask.py:163.

sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head d9f49af5 against base 20f3042b. CHANGES_REQUESTED — not merge-ready.

The earlier blockers remain resolved, and the far-ahead single-generation case works. The new skew_seen_at state is not scoped to the full current lease generation, however, which leaves two adjacent failures in the production writer.

Blocking — the first-sight bound is dropped when the clock catches up. _lease_state() consults skew_seen_at only while handed_at > now; afterward it returns to the original handoff deadline. With L=600, handed_at=T+599, and first observation at T, real clicks() records skew_seen_at=T without acting. At T+600 it is still non-actionable and due_clicks() is empty; it re-offers only at T+1199. That is 1,199 seconds after first sight, nearly two leases rather than the documented one.

The old observation also leaks into the fresh re-offer. The spread at lines 139–146 retains skew_seen_at. In an exact writer probe, a far-future handoff was first observed at T, then re-offered late at T+700; a one-second rollback to T+699 immediately produced needs_attention=True for that brand-new 600-second lease. A control that cleared the old marker recorded a new first sight and did not raise attention.

Please bind/reset the observation marker for each handoff/re-offer and keep the first-sight deadline effective even after the local clock passes handed_at. Add both regressions: a future offset of L-1, and a delayed re-offer followed by a one-second rollback. Worst case today is an extra lease of stranded recovery in one direction, or an immediate false persistent health warning and premature owner intervention in the other.

Evidence: exact-head card-click suite 44/44; all 42 focused pending-question/HITL entrypoints passed with two optional skips; exact-head CI and CLA are green; git diff --check and the review scanner are clean, with a firing hardcoded-path positive control. Those tests cover the submitted far-ahead and naive-expiry mutants but not these generation-boundary cases.

…r, not a wall-clock deadline

Replaces handed_at/lease_until/skew_seen_at with one invariant
(scripts/pending_questions_lease.py): a lease expires exactly when the time
credited to it reaches its window, and each pass credits
clamp(now - observed_at, 0, max_step), where observed_at only moves
forward. A backward step credits nothing and catching back up credits
nothing (no span counted twice), so no rollback can expire a lease or raise
NEEDS ATTENTION early; a forward jump credits at most LEASE_MAX_STEP_SEC
(300 s, the health check's fallback cadence); only deltas between passes
count, never a stamp's distance from this clock; a handoff or re-offer
starts a new lease. Every clicks pass and every clicks --due (the health
check) credits and saves the lease atomically under the clicks lock, so a
restart resumes from the saved credit.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@keweichen, replying to your review at d9f49af5: instead of a third boundary patch, the deadline and skew-marker logic is replaced by one invariant, at 58b4de3 (skills/pending-questions/scripts/pending_questions_lease.py, shared with #5947's pointer claim).

Invariant. A lease expires exactly when the time credited to it reaches its window. Each pass credits clamp(now - observed_at, 0, max_step), where observed_at is the latest wall time any pass has seen for that lease and only moves forward (observed_at = max(observed_at, now)).

Consequences:

  • A backward step credits 0. Catching back up to the earlier observed_at also credits 0, so no span is counted twice.
  • Repeated passes at one wall time credit 0.
  • A forward jump credits at most LEASE_MAX_STEP_SEC (300 s, the health check's launchd cadence) per pass. The click window is 600 s, so no single jump expires a lease.
  • Only deltas between passes count. A stamp's distance from this clock never matters.
  • A handoff or a re-offer starts a new lease {observed_at: now, elapsed: 0}, so nothing carries across generations.
  • Every clicks pass and every clicks --due (the health check) credits and saves the lease in the claim's one atomic store save, under the clicks lock. A restart resumes from the saved credit.

The trade-off I chose. observed_at only moving forward means a backward step of D seconds delays expiry by at most D, and never shortens it. Letting observed_at move back would double-credit the catch-up, which means stealing early. Your first finding (the stamp W−1 ahead) is now that case. After the clock steps back to T, nothing is credited until it passes the stamp, then one window of observed time runs, and there is no early expiry at any point. That is pinned by LeaseInProduction.test_a_stamp_just_under_a_window_ahead_is_held_one_window_of_observed_time.

Your second finding (a 1-second rollback after a late re-offer). The re-offer resets the lease, and the rollback credits 0. Result: needs_attention=False, lease {t+700, 0} (test_a_one_second_rollback_after_a_late_reoffer_never_raises_attention).

Tests:

  • LeaseInvariant table: W−1, W, W+1; a repeated time; a forward jump, then real deltas; a backward step; a 1-second rollback; rollback then catch-up with no double credit; renewal resets both fields.
  • Production writer: the two traces above; a far-future stamp; a real-process restart that resumes from the persisted lease; the lease starting at the handoff inside the lock.
  • Mutations: each of these fails 3–6 tests — moving observed_at back with the clock, crediting abs(delta), or removing the step cap.

card-clicks 45/45, plus every pending-questions and hitl suite and recovery-issues, pass.

@sonichi
sonichi requested a review from keweichen October 27, 2026 12:06

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 58b4de39e887421e2c9719626e5bb86e89624f9c against base 20f3042b821815ba9e3f191a8d69bedb6282bc56.

CHANGES_REQUESTED — not merge-ready. The renewal reset, rollback/catch-up, and persisted restart-credit cases requested last round now work, but two production transitions still return the wrong liveness/authority result.

  1. Blocking — a forward clock spike followed by correction freezes recovery for the entire spike. advance() caps the credited delta at 300 seconds but persists the full future wall value as observed_at. Every corrected pass below that high-water then credits zero. An exact production due_clicks() trace from a fresh handed claim produced:

    pass at T+1 year:       elapsed=300 due=False
    clock corrected to T:   elapsed=300 due=False
    corrected T+600:        elapsed=300 due=False
    corrected T+1 day:      elapsed=300 due=False
    T+1 year+300:           elapsed=600 due=True
    

    So a transient forward spike can strand a dead consumer's choice—with no re-offer or health warning—for a year (or any size spike). This regresses the prior far-ahead first-sight bound and contradicts the module's “neither … stretch” / “stamp distance never matters” contract. The test named test_a_far_future_stamp_does_not_matter sets observed_at=t, not a future stamp, so it cannot catch the failure. The submitted W−1 case also still expires at first sight + 1,199 seconds, exactly the trace requested last round. Please retain a per-generation bound from first local sight (or another restart-safe bounded clock design) and add true future-stamp plus forward-spike→correction production regressions.

  2. Blocking — a claim save rejected because the card became terminal still returns an actionable decision. stamp_claim() correctly reloads under the HITL store lock and skips a terminal card, but the handoff and re-offer callers ignore whether that transition landed. Meanwhile resolve() retires the card without the clicks lock. Deterministically injecting production resolve() immediately before each save gave this at both initial handoff and expired re-offer:

    resolve: decision card … retired
    card_status=cancelled
    actionable=True
    

    On re-offer, the fresh returned token was not persisted; the terminal card retained the old token. Worst case, the core executes a choice after the owner has terminally resolved the question, and the resulting token cannot even be acknowledged. Make the transition report whether it committed/current state and emit no actionable result after a rejected save (or serialize the whole authority transition), with resolve-vs-handoff and resolve-vs-re-offer regressions.

Evidence at this head: card-click suite 45/45; the three advertised naive lease mutants are caught (6 methods/12 subtest failures, 6/14, and 1/3 respectively), but none exercises spike→correction; review scanner passed with a firing hardcoded-path control; git diff --check is clean; all current CI and CLA checks are green. Those green checks miss the two traces above.

…ezes a lease; a retired card wins the claim save

- Lease: each pass credits clamp(now - observed_at, 0, max_step) and then
  sets observed_at = now. A spike of any size credits at most one step and
  normal time resumes once the clock is corrected (it no longer freezes the
  lease for the whole spike). The trade-off, stated in the module: catching
  back up after a rollback is credited again, a bounded over-credit of at
  most one step, never an unbounded freeze. A rollback still credits 0 and
  no single pass expires a fresh lease (max_step < window).
- stamp_claim reports whether the save landed; it is refused for a card a
  concurrent resolve() retired. The handoff, the re-offer and the first
  claim then hand nothing over, return no token, and re-read the card
  ("settled meanwhile — not a decision").

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@keweichen, replying to your review at 58b4de39: both findings are fixed at 6cf31ae. Each new test fails on 58b4de3.

1. A corrected forward spike froze recovery.

  • Fix: each pass now credits clamp(now - observed_at, 0, max_step) and then sets observed_at = now, with max_step < window (click: 300 s of a 600 s window).
  • What that guarantees:
    • A forward spike of any size credits at most one step. Once the clock is corrected, normal time resumes.
    • A rollback credits 0, so it never expires a lease or raises NEEDS ATTENTION.
    • Repeated passes at one time credit 0.
    • No single pass expires a fresh lease.
    • Renewal resets both fields.

This reverses your earlier "no double credit" pin, deliberately. Refusing all double credit means keeping observed_at at its high water, and that is exactly what freezes the lease for the whole span of a corrected spike: unbounded, as your trace shows. The cost now is that catching back up after a rollback is credited again, so a rollback can over-credit by at most one step (300 s). That bounded over-credit beats an unbounded freeze, and it never lets one pass expire a fresh lease. The trade-off is written in the module docstring, SKILL.md and the PR body.

Tests:

  • Table: a year's spike, then correction (correction credits 0, expired one pass later); rollback then catch-up (over-credit exactly one step); a 1-second rollback (over-credit 1 s); plus the earlier W−1/W/W+1, repeated-time and backward rows.
  • Spike sizes: spikes from an hour to 10¹² s each recover within window/step + 1 passes of normal time.
  • Production due_clicks(), your trace: (300, not due) at T+1 year, (300, not due) corrected to T, (600, due) at T+300.
  • Future stamps:
    • The W−1-ahead stamp now expires at first sight + 600 s.
    • A true year-ahead stamp is not taken on first sight.

2. A retired card still produced an actionable decision.

  • Fix: stamp_claim now reports whether the save landed. It refuses a card a concurrent resolve() has made terminal. On a refused save, the handoff, the re-offer and the first claim hand nothing over, return no token, and re-read the card. The output line reads "settled meanwhile — not a decision".
  • Tests (in RetirementRace): production resolve() is injected immediately before the handed save, at both the initial handoff and an expired re-offer. The card ends cancelled, actionable=False, no token is returned, and on the re-offer the stored token is still the old one.

card-clicks 50/50; every pending-questions and hitl suite, plus recovery-issues, passes. Changed-line coverage is 99.0%.

@sonichi
sonichi requested a review from keweichen October 27, 2026 12:48

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 6cf31ae9e888bdb2429173c2475c4d132dc9df92 against base 20f3042b821815ba9e3f191a8d69bedb6282bc56.

APPROVE for code. The prior two blockers are resolved. This PR is not merge-ready until the required CI rerun is green.

The shared lease now credits clamp(now - observed_at, 0, max_step) and records this pass's now, so a corrected forward spike resumes bounded progress. The production suite covers W−1/W/W+1, repeated timestamps, rollback/catch-up, a year spike followed by correction, a true future stamp, renewal, and persisted restart state. The deliberate downside is bounded catch-up over-credit: repeated clock oscillations can accumulate credit, but each pass is capped below the window and cannot expire a fresh lease alone. That tradeoff is documented and acceptable.

The terminal race is also fixed: the store-locked claim save reports whether it landed after re-reading the card, and initial handoff/re-offer expose neither an actionable result nor a token when production resolve() wins. Both injected races leave the card terminal; re-offer retains only the old persisted token.

Evidence at this head:

  • tests/pending-questions-card-clicks.test.py: 50/50
  • Seven current regressions run against prior head 58b4de39: 11 failures, covering spike correction, W−1 recovery, and refused-save races
  • High-water observed_at mutant: 11 failure records
  • Review scanner and git diff --check: clean

I read the current red CI. python standalone tests (2/9) timed out tests/hermetic-env-structural.test.py after 240 seconds; it did not report a product assertion failure, and the aggregate gate is red only as a consequence. That exact test passes locally 27/27 in 61s, and the same shard passes on cumulative child #5947. This looks like runner timing rather than a diff defect, but required CI still needs a green rerun before merge.

Nonblocking documentation nit: src/pending_questions_reader.py / the skill still call clicks --due read-only even though it intentionally persists lease credit.

@sonichi

sonichi commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

CI at 6cf31ae (run 40505837004): shard 2/9 failed on a timeout, not an assertion: TIMED OUT (>240s): tests/hermetic-env-structural.test.py. "tsc + tests (clean install)" was red only because that shard failed.

I believe it's a runner-speed flake, not this PR:

I re-ran the failed jobs without pushing, so the approved head is unchanged. The re-run passed, and run 40505837004 is now success.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 6cf31ae (agent review). The suite passes 50/50 and CI is green.

Verified by repro:

  • The year-ahead handoff re-offers after 3 passes (15 min).
  • A clock set back an hour never makes a live lease actionable.
  • A corrected spike costs one step.
  • A resolve injected at the handoff leaves the click unactionable and marked retired.

Non-blocking: over-credit is bounded per rollback, not in total. A ±1000 s flapping clock expired a live lease in 4 passes (4 s). The at-least-once rule covers it; a cap would tighten it.

@sonichi
sonichi merged commit 43b3c7c into main Oct 27, 2026
57 of 59 checks passed
@sonichi
sonichi deleted the feat/pq-card-clicks branch October 27, 2026 13:38
sonichi added a commit that referenced this pull request Oct 27, 2026
…lt or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 27, 2026
… never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Oct 28, 2026
…lt or --option is given (#5947)

* feat(pending-questions): ask-owner asks by decision card when --default or --option is given

The ask side of decision cards, stacked on the click side (#5946), split
out of #5942.

- src/hitl/cards.py: one card builder (choice_request, create_choice,
  manager_for, new_id); report-feedback and the worker pool's wedge cards
  build through it, and the clicks module takes its store handle from it.
- queue_question builds the card with a pre-allocated id, holds the
  question (outbox, then the row) naming it, and only then stores the card.
  Once a card exists no text is sent. create_card returns stored / not
  stored / unknown (or no card); stored and unknown keep the card and report
  that no text was sent.
- A definite card failure texts the question to its own destination with
  the row's own link (the one the row write returned), then holds the
  question again flagged `redeliver`; a landed row is rewritten now
  (RoomDbStore.redeliver) and the entry released, and an unrewritten or
  unconfirmed row takes the record at the next reconcile (Outbox.flush).
- The reminder never texts a question whose card is live, --force included;
  recency is the gateway's notified_at once posted; a card unposted after
  30 min gets one text pointer, once per card, with no options.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): claim the card pointer before publishing it; commit history before releasing a fallback hold

- send_card_pointers now claims each pointer first (claim_pointer: under the
  HITL store lock, re-check the card is live, unposted, past the grace and
  unclaimed, then stamp pointer_at with the store's fsynced save) and only
  the claimant publishes. Overlapping reminders queue one pointer; a death
  after the claim queues none; a card posted before the claim gets none.
- The definite-failure fast path now commits store history, then rewrites
  the landed row, then releases the redeliver hold (the order Outbox.flush
  keeps). A history that cannot be written keeps the hold, so a later room
  outage lists the question instead of a measured zero.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): a card pointer is claimed, published, then marked sent; a stale claim is retried

The pointer state is split: claim_pointer stamps pointer_claim_at and a
token under the HITL store lock, the claimant publishes, then
mark_pointer_sent records pointer_sent_at. A claim with no sent mark older
than POINTER_CLAIM_RETRY_SEC (5 min) — a claimant that died or failed to
publish — is claimed again and retried. The pointer is now at-least-once
(a death between the publish and the sent mark can deliver it twice), and
SKILL.md says so.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): a pointer is re-checked before it is written, never revives a settled card, and a back-stepped clock cannot steal its claim

- send_card_pointers re-checks the card under the store lock just before
  writing the file (live, unposted, still this claim's) and writes it inside
  that lock, so a post recorded after the claim cancels the pointer. The
  window left is a projector send in flight (it records the post only after
  its send returns): bounded by one send, documented.
- mark_pointer_sent refuses a terminal record, so a card settled or archived
  after the claim is never written back live.
- A pointer claim stamped in the future (the clock went back) stays fresh
  until its window passes, so a second claimant cannot take it.
- The card-record reader (card_sent / card_state / card_live) now lives in
  the parent (#5946); this layer keeps the pointer and sent_at logic.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): a pointer claim stamped ahead of this clock holds one window from first sight

Same bound as the click lease: a pointer_claim_at ahead of this clock is
never taken on sight (a backward step cannot steal a fresh claim), but the
first reminder that sees it records pointer_skew_seen_at and the claim can
be retried one POINTER_CLAIM_RETRY_SEC after that, so a clock that ran far
ahead no longer suppresses the pointer for the whole skew.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): a pointer claim uses the same elapsed-time lease as a click

Replaces pointer_claim_at / pointer_skew_seen_at with the shared lease
(pending_questions_lease): a pointer claim is kept until its lease has been
credited POINTER_CLAIM_RETRY_SEC across reminder passes, each pass crediting
clamp(now - observed_at, 0, POINTER_MAX_STEP_SEC) with observed_at only
moving forward; a new claim starts a new lease. No clock step replaces a
live claim, and a near- or far-future stamp holds nothing beyond the
observed window.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pending-questions): pin the pointer claim lease under rollback over-credit and a corrected year spike

The parent's lease now sets observed_at = now each pass (a corrected forward
spike resumes normal time instead of freezing); the pointer claim table
pins one step, W-1, a 1 s rollback, the bounded catch-up over-credit, a
renewal, a year's spike then correction, and recovery after it.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): every reminder pass credits a pointer claim's lease; grace only gates the replacement

claim_pointer returned before crediting an existing claim when the card was
still inside its grace, so passes before grace never counted. It now
credits and saves the existing lease on every pass first, returns without
replacing while still before grace, and mints a replacement only when the
lease has expired AND the card is grace-eligible.

Also: `clicks --due` is described as writing lease credit (not read-only),
and the trailing blank line at the end of src/hitl/cards.py is gone.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pending-questions): send the unposted-card pointer on every pass, flagless included

The 30-minute text pointer for a decision card the gateway never posted ran
only under --notify, which is on demand, so an unattended host never sent it.
The pointer completes the question's one ask rather than reminding, so it now
runs on every pass of src/check-pending-questions.py (the proactive loop's
flagless pass included), outside presenter mode; text reminders still need
--notify. Docs (CLAUDE.md/AGENTS.md, SKILL.md, proactive-loop, rationale) say
so, and the crash-before-hold text is now attributed to the next --notify run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(proactive-loop): keep step 2 under the skill byte cap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pending-questions): replace source-text assertions with behaviour tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants