Repository navigation
health-check: auto-restart ngrok and conversation-server - #207
Merged
Merged
Conversation
ngrok and conversation-server are critical for phone calls but had no auto-restart logic. When ngrok dies (e.g. when conversation-server is restarted, ngrok sometimes loses its tunnel), inbound calls return "busy" until manually restored. Changes: - ngrok status now "down" (was "warn") when missing — eligible for fix - Added fix logic for ngrok (uses NGROK_DOMAIN env var if set) - Added fix logic for conversation-server The proactive loop calls health-check.py --fix when issues are found, so this enables self-healing for the phone stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sonichi
added a commit
that referenced
this pull request
Apr 8, 2026
PR #207 added auto-restart for ngrok, but MacBook uses Tailscale Funnel instead. This makes the tunnel check tunnel-agnostic: - If TWILIO_WEBHOOK_URL contains "ts.net", check the Funnel by curling the /health endpoint over the public URL (verifies end-to-end) - Otherwise check ngrok via port 4040 (existing behavior) - Auto-restart adds tailscale-funnel handler that runs \`tailscale funnel --bg 3100\` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sonichi
added a commit
that referenced
this pull request
Apr 8, 2026
PR #207 added auto-restart for ngrok, but MacBook uses Tailscale Funnel instead. This makes the tunnel check tunnel-agnostic: - If TWILIO_WEBHOOK_URL contains "ts.net", check the Funnel by curling the /health endpoint over the public URL (verifies end-to-end) - Otherwise check ngrok via port 4040 (existing behavior) - Auto-restart adds tailscale-funnel handler that runs \`tailscale funnel --bg 3100\` Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sonichi
pushed a commit
that referenced
this pull request
Apr 8, 2026
health-check.py's auto-restart path (PR #207) already reads NGROK_DOMAIN and passes --domain=... so the tunnel comes back on the same URL. But startup.sh still ran plain 'ngrok http 3100', so the initial tunnel URL was random and the Twilio webhook had to be updated manually on every first boot. Mirror the health-check logic in startup.sh: read NGROK_DOMAIN from .env, pass --domain=... if set, otherwise fall back to the old random-URL behaviour. Document the variable in .env.example. Motivated by the Mac Mini migration — ngrok reserved domain was set in .env but startup.sh ignored it, so the first boot after reboot still printed "⚠ Update Twilio webhook to: ...". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 of 3 tasks
sonichi
added a commit
that referenced
this pull request
Apr 8, 2026
health-check.py's auto-restart path (PR #207) already reads NGROK_DOMAIN and passes --domain=... so the tunnel comes back on the same URL. But startup.sh still ran plain 'ngrok http 3100', so the initial tunnel URL was random and the Twilio webhook had to be updated manually on every first boot. Mirror the health-check logic in startup.sh: read NGROK_DOMAIN from .env, pass --domain=... if set, otherwise fall back to the old random-URL behaviour. Document the variable in .env.example. Motivated by the Mac Mini migration — ngrok reserved domain was set in .env but startup.sh ignored it, so the first boot after reboot still printed "⚠ Update Twilio webhook to: ...". Co-authored-by: Chi <wangchi@Chis-Mac-mini.local> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
liususan091219
pushed a commit
to liususan091219/sutando-fork
that referenced
this pull request
May 28, 2026
…nichi#207) ngrok and conversation-server are critical for phone calls but had no auto-restart logic. When ngrok dies (e.g. when conversation-server is restarted, ngrok sometimes loses its tunnel), inbound calls return "busy" until manually restored. Changes: - ngrok status now "down" (was "warn") when missing — eligible for fix - Added fix logic for ngrok (uses NGROK_DOMAIN env var if set) - Added fix logic for conversation-server The proactive loop calls health-check.py --fix when issues are found, so this enables self-healing for the phone stack. Co-authored-by: Chi <wangchi@Chis-Mac-mini.local> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
liususan091219
pushed a commit
to liususan091219/sutando-fork
that referenced
this pull request
May 28, 2026
PR sonichi#207 added auto-restart for ngrok, but MacBook uses Tailscale Funnel instead. This makes the tunnel check tunnel-agnostic: - If TWILIO_WEBHOOK_URL contains "ts.net", check the Funnel by curling the /health endpoint over the public URL (verifies end-to-end) - Otherwise check ngrok via port 4040 (existing behavior) - Auto-restart adds tailscale-funnel handler that runs \`tailscale funnel --bg 3100\` Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
liususan091219
pushed a commit
to liususan091219/sutando-fork
that referenced
this pull request
May 28, 2026
health-check.py's auto-restart path (PR sonichi#207) already reads NGROK_DOMAIN and passes --domain=... so the tunnel comes back on the same URL. But startup.sh still ran plain 'ngrok http 3100', so the initial tunnel URL was random and the Twilio webhook had to be updated manually on every first boot. Mirror the health-check logic in startup.sh: read NGROK_DOMAIN from .env, pass --domain=... if set, otherwise fall back to the old random-URL behaviour. Document the variable in .env.example. Motivated by the Mac Mini migration — ngrok reserved domain was set in .env but startup.sh ignored it, so the first boot after reboot still printed "⚠ Update Twilio webhook to: ...". Co-authored-by: Chi <wangchi@Chis-Mac-mini.local> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qingyun-wu
added a commit
that referenced
this pull request
Jul 26, 2026
…ified, approval consumed once capability.execute now executes: message.send posts through the gateway (daemon-resolved REMOTE_TASK_URL/REMOTE_TASK_TOKEN — the CLI can never supply credentials) and only a response carrying the posted message's event_id counts as delivered (the #2324/#207 fail-closed contract); a swallowed-send 200 fails the request. --approval <requestId> gates an execution on an APPROVED, unconsumed approval and consumes it atomically — one approval authorizes exactly one execution. Unknown actions fail cleanly. E2E grows to 16 checks against a mock gateway speaking the broker contract: approved→execute delivers with event_id; second execute on the same approval refuses (already consumed); 200-without-event_id fails closed; unknown action fails.
qingyun-wu
added a commit
that referenced
this pull request
Jul 26, 2026
… + CLI + durable store (v0) (#2325) * feat(runtime-api): local approval/elicitation/capability RPC — daemon + CLI + durable store (v0, E2E-tested) The request-response upstream between the long-running agent and its human owner, per the sutando-socket design (workspace notes): a local daemon on a private Unix socket speaking NDJSON JSON-RPC, a sutando-runtime CLI so the agent never touches sockets or remote APIs, and a SQLite request store so every request is durable, resolves exactly once (CAS transitions, terminal states immutable), and survives daemon restarts. v0 scope decision: approval/elicitation ride the EXISTING human-action card lifecycle (ha_adapter mirrors a request into an ActionStore pending action → the gateway CardPoster posts the buttons card → DecisionHandler routes the owner's answer back → the resolver loop maps it onto the request's terminal state). No new server API or UI — the formal /v1/agent-requests can come later without touching this contract. capability.execute is a fake executor recording the full lifecycle; slice 2 replaces it with the governed gateway send verified by event_id. Identity is daemon-resolved from env (never CLI-self-reported); socket dir 0700 / socket 0600; live-daemon probe before stale-socket takeover; 256KB frame cap; issuing methods never block on a human (request.wait carries the bounded blocking). E2E (tests/runtime-api-e2e.test.py, stdlib, real daemon + real CLI + real ActionStore; only the owner's answer is simulated in DecisionHandler's exact write shape): approve, deny, single_select elicitation, capability, unknown-id error, wait-timeout, restart recovery, cancel-beats-late-answer — 12/12 green. * fix(runtime-api): sanctioned repo-root resolution + src-map regeneration lint-workspace-resolution flagged the adapter's parent-chain repo walk: sibling-import bootstrap now goes through named intermediates and repo root comes from sutando_config._find_repo_root (marker walk), never a bare parent chain. docs/src-map.md regenerated for the new modules. E2E re-run green (12/12). * feat(runtime-api): slice 2 — real message.send executor, event_id-verified, approval consumed once capability.execute now executes: message.send posts through the gateway (daemon-resolved REMOTE_TASK_URL/REMOTE_TASK_TOKEN — the CLI can never supply credentials) and only a response carrying the posted message's event_id counts as delivered (the #2324/#207 fail-closed contract); a swallowed-send 200 fails the request. --approval <requestId> gates an execution on an APPROVED, unconsumed approval and consumes it atomically — one approval authorizes exactly one execution. Unknown actions fail cleanly. E2E grows to 16 checks against a mock gateway speaking the broker contract: approved→execute delivers with event_id; second execute on the same approval refuses (already consumed); 200-without-event_id fails closed; unknown action fails. * fix(runtime-api): ha action ids must be hex-only — live human acceptance caught it The first real owner answer could not resolve: DecisionHandler's answer grammar matches ha_[0-9a-f]{6,} (hex only) and the adapter's id carried the requestId's type prefix ('ha_approval...'). The local E2E missed it by writing resolutions directly through ActionStore — exactly the gap a real-human test exposes. Ids now derive from the requestId's hex uuid tail (shared ha_action_id helper, used by issue + recovery), and a regression pins the id against the answer grammar itself. * fix(runtime-api): close the three review P1s — governed gating, idempotency, elicitation dead paths 1. message.send (and any GOVERNED_ACTIONS entry) now REQUIRES an approved, unconsumed approval, validated and consumed before the executor can touch daemon-held credentials — an ungated socket client is refused pre-gateway. Regression pins that no gateway POST happens on the refused call. 2. idempotencyKey is persisted (unique-indexed column) before approval consumption and gateway contact; a retry with the same key returns the recorded request (idempotentReplay: true) — no duplicate send, no spurious already-consumed failure. Regression: two identical executes → one gateway POST, same requestId. 3. Elicitation dead paths: multi_select now sets the multiSelect flag so DecisionHandler's comma grammar resolves it (E2E covered); free_text is rejected loudly in v0 (the card transport is options-based — a stranded-until-expiry request is worse than a clear error); single/multi_select require options; default type is single_select. E2E grows to 24 checks, all green. * fix(runtime-api): four review P1s — migration, approval binding, key fingerprint, non-blocking executor 1. Rolling-update migration: a pre-idempotency DB (the live acceptance created one) gains the new columns via idempotent ALTER before the index script — no boot crash. Regression boots the old schema and reads its rows. 2. Approval-capability BINDING: an approval authorizes exactly the action+resource the owner saw on the card, compared canonically BEFORE consumption — an approved repo.force_push can no longer authorize a message.send, and a mismatch costs nothing. Regressions: cross-action and cross-resource both refused. 3. Idempotency-key fingerprint: the key persists a sha256 of action+resource+input; reuse with a different fingerprint is rejected instead of replaying an unrelated side effect as complete. 4. Blocking executors off the event loop (asyncio.to_thread): a slow gateway send no longer stalls other clients or the resolver — regression drives a 1.5s send concurrently with a request.get that must return <1s (measured 0.05s). E2E now 27 checks, all green. * test(runtime-api): in-process RequestStore unit suite — coverage-visible lifecycle pinning The E2E drives the store through the daemon subprocess, invisible to the coverage recorder — the diff-coverage gate flagged request_store at 63.8% the moment the migration regression made the file visible. This suite pins the durable-lifecycle contract in-process: create/get roundtrip, unknown-id, pending listing, CAS transition + terminal immutability, one-time consumption (approved-only, unknown refused), lazy durable expiry, result persistence, idempotency-key lookup with fingerprint, and the pre-idempotency schema migration (readable old rows, usable index, writable new columns, clean re-boot). 20 checks. * fix(runtime-api): record creation + approval consumption are one atomic transaction create_consuming() inserts the capability record (carrying its idempotency key + fingerprint) and stamps the approval consumed in a single SQLite transaction — a crash between the steps can no longer spend an approval without leaving a replayable record, and a same-key/ different-approval race rolls back untouched when the unique index rejects the insert (the loser replays the winner's row). Also: CLI elicitation --type default free_text → single_select (the v0 server rejects free_text; the default path should not be an intentionally unsupported request). Regressions: store suite pins atomic commit/rollback semantics (no orphan row on a lost consume race; duplicate key leaves the second approval consumable); E2E adds same-key+different-approval replay with the fresh approval provably still spendable. * fix(runtime-api): shared cross-platform run-dir helper; _issue never strands a pending row Blocker 1: the daemon and CLI each duplicated an invented macOS-only run-dir fallback. rundir.py is now the ONE policy both import (SUTANDO_RUN_DIR > darwin app-support run dir > XDG_RUNTIME_DIR/sutando > ~/.sutando/run), so the default socket resolves identically on both ends of the connection, on every platform. Blocker 2: _issue committed the durable pending row before the human-action mirror; an open_approval/open_elicitation failure left an unanswerable pending request while the API errored. The mirror call is now guarded — on failure the row is durably marked failed (error recorded) before the ProtocolError surfaces, and restart recovery re-links nothing for it. In-process failure-injection regression (tests/runtime-api-issue-failure.test.py, 5 checks) pins both the failure and success paths. * ci(runtime-api): rundir owner allowlisted; src-map regenerated; rundir branch coverage Three CI gates from the rundir.py introduction: (1) the install-path lint now allowlists src/runtime-api/rundir.py as the ONE owner of the runtime run-dir fallback (the pattern the lint prescribes — daemon + CLI import it); (2) docs/src-map.md regenerated for the new module; (3) in-process unit suite pins every resolution branch (env override, darwin, XDG, portable fallback, socket override + derived default) so the diff- coverage gate sees the module the subprocess daemon hides. * ci: allowlist the rundir policy test (pins the owner file's fallback literal) tests/runtime-api-rundir.test.py asserts rundir.py's documented ~/.sutando/run fallback — the same pin-the-owner's-policy role tests/workspace-default.test.py already plays in the allowlist. (The previous local lint pass missed it: diff-mode scans committed files and the test was still untracked.) * test(runtime-api): in-process daemon suite — coverage-visible server/protocol/ha_adapter The E2E proves behavior through a subprocess daemon the coverage recorder cannot see (same gap the store suite closed for request_store.py). This suite drives the daemon internals in-process: the protocol codec (all reject branches), issue + elicitation validation, the full capability path (governed gating, binding, idempotent replay, the duplicate-key race branch both with and without a winner row), settle mapping (approve/deny/elicitation answers incl. single_select index flattening, expiry), resolver_loop error isolation, the live Unix-socket serve() loop (round trip, malformed frame, oversized-frame disconnect, -32000 on a non-protocol handler crash, live-socket takeover refusal), main() env wiring incl. KeyboardInterrupt, the real message.send executor against an in-process HTTP stub (delivered / swallowed-200 / bad input / unconfigured), and the ha adapter's remaining branches. Gate-measured locally (.coveragerc, per-file runs + combine): server 99% · ha_adapter 99% · protocol 100% · request_store 100% · rundir 100%. * fix(runtime-api): approval binds the exact effect incl. input; restart never strands a consumed-approval row Blocker 1 (input substitution): binding compared only action+resource while message.send takes its payload from params.input — a caller could obtain approval for a benign body and execute any other body. The card now SHOWS the governed input (open_approval Input: line), the CLI's approval request accepts --input, and binding compares the same canonical action/resource/input fingerprint idempotency uses — checked BEFORE consumption, so a substituted input is refused at zero cost and the approval stays spendable for the exact approved effect. Blocker 2 (restart strand): create_consuming() commits the pending row and spends the approval before the executor runs; recover() previously re-linked only approval/elicitation rows, leaving a crash-window capability row pending forever with its one-time approval consumed. recover() now transitions such rows to an honest terminal failed state (outcome recorded as unknown; no silent replay that could double-send; retry requires a fresh approval by design). Regressions: E2E — card shows the body, substituted input refused pre-gateway with the approval still spendable (31 checks); in-process — same pair plus the simulated crash window: row pending + approval consumed → fresh server recover() → failed('interrupted by daemon restart'), approval stays spent, wait returns failed not a pending timeout. Coverage (gate methodology): server 99 · ha_adapter 99 · protocol 100 · store 100 · rundir 100. * fix(runtime-api): recovery reports outcome UNKNOWN, never executed:false The crash window spans both sides of the gateway send: recovery cannot know whether the effect landed. Asserting executed:false invited a confident duplicate retry with a fresh approval (review P1 — the exact double-send the no-silent-replay design exists to prevent). The recovery result now carries outcome:'unknown' with an explicit verify-before-retry instruction and no executed boolean at all; the regression pins both the field and the boolean's absence. * chore(runtime-api): English-only comments/docstrings (public repo) --------- Co-authored-by: Qingyun Wu <qingyun@ag2.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
python3 src/health-check.py --fix, verify it restarts🤖 Generated with Claude Code