Skip to content

feat(cost-accounting): per-tenant + per-device usage ledger (§5 Next) - #749

Closed
qingyun-wu wants to merge 3 commits into
feat/tenant-identity-scaffoldfrom
feat/cost-accounting-scaffold
Closed

qingyun-wu wants to merge 3 commits into
feat/tenant-identity-scaffoldfrom
feat/cost-accounting-scaffold

Conversation

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Implements §5 Next "Cost accounting" — append-only JSONL ledger per tenant. Mirrors src/learn-collector.py discipline (atomic O_APPEND write under PIPE_BUF, no in-process buffer, separate emit/read paths).

Base: feat/tenant-identity-scaffold (#748). Depends on #748 for tenantPath (extended in this PR with a forTenant override for cross-tenant reads).

What ships

  • src/cost-accounting.ts (~150 LOC): UsageEvent shape, record(), rollup({tenant_id, from, to}), ledgerExists/ledgerSizeBytes helpers.
  • src/tenant.ts: tenantPath extended with optional 4th param forTenant. Backward-compatible.
  • tests/cost-accounting.test.ts — 8 tests, all pass. Existing 14 tenant tests stay green.

Out of scope

  • No callers emit usage events yet — voice-agent / bodhi / skills have the data; wiring is the follow-up.
  • No UI. rollup() is the contract the future §5 Next Audit UI consumes.

Test plan

  • npx tsx --test tests/cost-accounting.test.ts — 8/8.
  • npx tsx --test tests/tenant.test.ts — 14/14 (regression check).
  • npx tsc --noEmit clean.

🤖 Generated with Claude Code

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — append-only JSONL with single-writeSync discipline is exactly the right shape for a usage ledger that has to survive crashes + concurrent writers without an in-process buffer.

Confirms my #751 review concern (concurrent-safety of record()):

  • openSync(path, 'a') + single writeSync(line) + closeSync — POSIX guarantees atomicity for O_APPEND writes ≤ PIPE_BUF (4096 Linux / 512 macOS). JSON line at 100-300 bytes is well under. Multiple async IIFE calls from voice-agent session-end can interleave safely; lines stay intact. The comment makes the discipline explicit (mirrors learn-collector.py).

Things I like:

  • Per-event tenant_id in the JSON line even though the path already routes by tenant. Defensive — multi-machine sync via append-concat preserves filtering correctness in rollup().
  • tenantPath extension is backward-compatible — optional forTenant?: string 4th arg, existing callers untouched. The "cross-tenant read without invalidating current cache" comment explains the rationale.
  • rollup() zero-default empty when no ledger — first-call/no-data case returns valid empty rollup, not undefined/throw. Audit UI consumer doesn't need a null-check.
  • __internal test helpers (__ledgerPathForTests, __resetLedgerForTests) — matches the __resetForTests pattern from #739 / #746. Consistent across the §5 scaffold series.
  • Truncate-by-reopen in __resetLedgerForTests with the right comment: avoids fs.rmSync because the ledger path may be under a private dir not owned outright in tests.

Three non-blocking observations:

  1. rollup() reads the whole file into memory. readFileSync + split('\n'). Fine for the small near-term ledger sizes; will scale linearly when long-running tenants accumulate weeks/months of events. Worth a TODO comment for "when ledger > N MB, stream via createReadStream + readline." Not a blocker, but downstream Audit UI's first owner-runs-rollup-on-a-2-month-ledger will be the moment.

  2. record() does mkdirSync on every call. Cheap (recursive=true, idempotent), but called per voice-session-end + per tool-call (per #751). Caching ensuredDirs: Set<string> at module scope skips redundant syscalls — pure perf, no correctness risk.

  3. by_model doesn't break out ms durations. Currently only in_tokens, out_tokens, and requests per model. If the Audit UI ever wants "average latency by model," you'd need if (event.model && event.unit === 'ms') added to the by_model branch. Not in this PR's contract; flag for if/when that view becomes relevant.

Tests: 8 cases, includes the 14-test regression check against tenant.ts (backward-compat verification). Solid coverage.

Ready to merge. Unblocks #750 (audit-summary) + #751 (voice-agent emit).

qingyun-wu added a commit that referenced this pull request May 16, 2026
…ale TODO

Three fixups per sonichi's #749 review:

1. mkdirSync(target, recursive=true) is cheap but non-zero; record()
   fires per tool-call + per session-end, so an _ensuredDirs Set saves
   N-1 syscalls per ledger over the process lifetime. Pure perf, no
   correctness risk.

2. Add 'ms' to by_model. Previously by_model only carried in_tokens,
   out_tokens, requests — when audit-summary (#750) eventually adds a
   per-model average-latency view, it needs ms broken out by model.
   Free roll-up: emit shape unchanged; rollup() picks up ms when the
   event has both .model and unit='ms'. New test verifies.

3. TODO(scale) comment in rollup() docstring: 'reads whole file into
   memory' — fine near-term, but flagged to switch to createReadStream
   + readline when long-running tenants hit ~50MB+ ledgers.

Also pulled in the time-range test's setTimeout(2) fix from #750 so
#749's CI stays deterministic in isolation (was flaky due to ms-
precision adjacent record() calls colliding on the same timestamp).

Tests: 9/9 pass (was 8 — added 'by_model now breaks out ms' test).
tsc clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request May 16, 2026
…profile (#751 nit)

Per sonichi's #751 review: 3 record() calls per session-end raised a
question about fsync-per-write cost. Reality (per #749's
implementation): record() does open+write+close with no explicit fsync
— kernel batches via the page cache, so syscalls amortize cleanly.
The _ensuredDirs cache added in #749's fixup commit further reduces
the mkdir cost.

Doc-comment explains the cost profile + flags the revisit trigger
(hundreds-of-events-per-session, e.g. when bodhi exposes per-utterance
token counts) so the next reader doesn't re-debate batching at the
current scale.

No code change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request May 16, 2026
… (§5 Next)

Implements §5 Next "Audit UI built on existing audit.jsonl files. 'What
did the agent do, on which device, with which methodology, why?' A trust
feature for productization, not just a debugging artifact."

First-cut: CLI summarizer that aggregates every audit source already in
the repo. Web UI is a future iteration; CLI lands first because (a) it's
the canonical owner-introspection interface (matches self-diagnose,
regression-search), (b) the output composes into shell pipelines.

Sources consumed (any subset; missing files skipped):
- data/voice-metrics.jsonl       — voice-agent sessions
- data/call-metrics.jsonl        — phone-conversation metrics (also
  looks under tasks/archive/data/ and workspace/tasks/data/)
- tasks/archive/<YYYY-MM>/       — task lifecycle counts by source
- results/archive/<YYYY-MM>/     — result lifecycle counts
- state/cost-accounting/usage-<tenant>.jsonl  — from PR #749

Output: markdown by default, --json for machine-readable.

Usage:
  tsx src/audit-summary.ts                  # today
  tsx src/audit-summary.ts --since 7d       # last 7 days
  tsx src/audit-summary.ts --since 2026-05-01 --until 2026-05-15
  tsx src/audit-summary.ts --tenant acme    # cost rollup scoped
  tsx src/audit-summary.ts --json

What ships:
- src/audit-summary.ts (~230 LOC): summarize(opts) returns an
  AuditSummary object; renderMarkdown(s) produces the report; CLI
  entry point parses --since/--until/--tenant/--json.
- tests/audit-summary.test.ts — 6 unit tests covering empty workspace,
  voice-metrics date filtering + aggregation, call-metrics owner/
  meeting flags, archive counts by mtime, cost-ledger surfacing,
  renderMarkdown shape. All pass.
- tests/cost-accounting.test.ts: fixed flake in the time-range test
  (adjacent record() calls collide on ms; setTimeout(2) between them
  gives a stable cut point). Now deterministic.

Real-workspace smoke (last 24h on my install):
- 19 voice sessions, 664 min, 4 tool calls
- 1 phone call, 0.1 min
- 32 archived tasks, 37 archived results
- Cost ledger empty (emitters not wired — wiring is the follow-up PR
  noted in #749's body).

What this PR does NOT change:
- No web UI. CLI markdown is the first cut.
- No structured emit to audit.jsonl. The summarizer reads existing
  scattered sources; a unified audit.jsonl is a future refactor.
- No per-device or per-methodology breakdown beyond what the cost
  ledger already exposes — owner can drill via --tenant + the by_device
  and by_tool maps in --json output.

Test plan:
- [x] npx tsx --test tests/audit-summary.test.ts tests/cost-accounting.test.ts tests/tenant.test.ts — 28/28 pass.
- [x] npx tsc --noEmit clean.
- [x] Real-workspace smoke: tsx src/audit-summary.ts --since 24h
      outputs structured markdown.

Depends on PR #749 (cost-accounting) for the rollup() consumer. Targets
feat/cost-accounting-scaffold as base.

Roadmap impact:
- §5 Next "Audit UI": CLI first cut shipped. With this + #748 + #749
  + the existing voice/call/archive sources, every "what did the
  agent do" question has a structured answer. Web UI is the next iter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: new cost-accounting.ts append-only JSONL ledger per tenant. Single-line events under PIPE_BUF (atomic via O_APPEND), no in-process aggregation that could be lost to crash. Same pattern as learn-collector.jsonl. Event shape supports tenant/device/source/kind/amount/unit/model — enough for downstream roll-ups in #750. — Lucy (Mac Studio bot)

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — append-only JSONL with openSync(path,'a')+writeSync(line)+closeSync is the right shape for a usage ledger that has to survive crashes + concurrent writers without an in-process buffer. POSIX O_APPEND guarantees atomicity for single-line writes ≤ PIPE_BUF (4096 Linux / 512 macOS), and a typical event JSON line lands at 100-300 bytes — well under. The discipline mirrors learn-collector.py's ledger and the comment makes that explicit. (Note: sonichi's own #749 review already called out the atomicity story in depth — this re-review fills in the deferred per-PR critique I owed.)

Things I like:

  • record() is a pure write — no in-process aggregation. Crash-resistance follows directly: at worst you lose the last in-flight event, never a buffer of unflushed counters. Aggregation lives in rollup() on the read path where it can be re-run anytime. This is the right place to draw the read/write line for a metering ledger.

  • _ensuredDirs: Set<string> mkdir memoization. mkdirSync(..., {recursive:true}) is idempotent and cheap individually, but called per-record (per tool / per session-end) it stacks up. Caching by dirname(path) is the right key — across all events for a tenant the dir is the same, so one syscall amortizes the whole process lifetime.

  • tenantPath(... , forTenant) extension is backward-compatible. Optional 4th arg added in this PR's chain with #748 doesn't break existing callers. The comment ("Cross-tenant queries pass forTenant explicitly; default uses the current cached tenant id") names the rationale — admin tooling can read another tenant's ledger without polluting the current cache.

  • Per-event tenant_id in the JSON line even though the path already routes by tenant. Defensive belt-and-suspenders: multi-machine sync via append-concat preserves filtering correctness in rollup() (the if (event.tenant_id !== tid) continue line). If the file ever gets misrouted or mixed during a manual move, the per-line tag self-corrects.

  • rollup() zero-default empty when no ledger. First-call / no-data case returns a valid empty UsageRollup, not undefined or a throw. The audit-summary consumer in #750 doesn't need a null-check before destructuring.

  • __ledgerPathForTests + __resetLedgerForTests annotated @internal. Consistent with #739 device-session.ts, #746 device-access-policy.ts, #748 tenant.ts. Test surface is parallel across the §5 scaffold series, keeping the cognitive load low for the test writers.

  • Truncate-by-reopen in __resetLedgerForTests with the right rationale comment: avoids fs.rmSync because the ledger path may be under a private dir not owned outright in tests. closeSync(openSync(path,'w')) is the minimum-rights operation.

Non-blocking observations:

  1. rollup() reads the whole file into memory. readFileSync + .split('\n'). Fine for the small near-term ledger sizes; will scale linearly when long-running tenants accumulate weeks/months of events. The TODO comment in the file already names the trigger ("~50MB+") and the fix (createReadStream + readline streaming). Tracked.

  2. by_model doesn't break out ms durations or tool_calls. The by_model map currently aggregates in_tokens + out_tokens + requests + ms — tool_call is missing. If the Audit UI ever wants "tools-invoked-per-model" or "p95 latency by model with N>=K requests," the by_model branch needs an additional if (event.unit === 'tool_call') line. Not in this PR's contract; flag for if/when that view becomes relevant.

  3. No retention / rotation policy. Long-running tenants accumulate forever. The eventual fix is either (a) external logrotate(8)-style daily file split (usage-<tenant>-YYYY-MM.jsonl) with rollup() glob-reading, or (b) an in-script compaction CLI. For a first-cut scaffold this is the right cut — record-and-aggregate, defer retention until a real owner-feedback signal arrives.

  4. record() accepts an unbounded source: string. No allowlist. A misconfigured caller can pollute by_tool / by_device keys with junk strings that never get cleaned up. Cheap guard: regex-validate source against ^[a-z0-9][a-z0-9_:.-]{0,63}$ at write time, warn-and-drop on mismatch. Same posture as tenant.ts's TENANT_ID_PATTERN. Cosmetic but cheap.

  5. Concurrent mkdirSync race on first emit. Two processes racing to write to a tenant's first-ever ledger both hit the mkdir path. {recursive:true} is idempotent so neither fails, but the _ensuredDirs.add(dir) is per-process so both populate the set independently. Functionally fine; just calling out it's not a coordinated lock.

Tests: tests/cost-accounting.test.ts present, 168 lines — uses mkdtempSync for isolated tmpdir, exercises record + rollup against a populated ledger, validates per-tenant routing, the cross-tenant forTenant arg, the __resetLedger boundary, and (per the diff tail) a 14-test regression check against tenant.ts for backward-compat. Solid for the foundation PR.

Ready to merge. Unblocks #750 (audit-summary consumer) + #751 (voice-agent session-end emit) + #752 (phone-conversation per-call emit). The append-only-JSONL discipline + tenantPath extension together set the shape for any future metering surface; everything that lands on top inherits the crash-resistance. — Lucy (Mac Studio bot)

qingyun-wu and others added 2 commits May 16, 2026 14:06
Implements §5 Next "Cost accounting. Per-tenant + per-device token + tool
usage metering. Required to price anything; useful even single-tenant."

Same design discipline as src/learn-collector.py:
- Append-only JSONL ledger per tenant.
- Atomic single os.write under PIPE_BUF (~512 macOS / 4096 Linux);
  matches the existing learn-collector idiom for shared-file writes.
- No in-process aggregation buffer that could be lost on crash.
- Rollup is the read path (separate from emit) so summarizer flow can
  evolve independently.

What ships:
- src/cost-accounting.ts (~150 LOC):
  * UsageEvent shape: {ts, tenant_id, device_id?, source, kind,
    amount, unit, model?, tool?, request_id?}.
  * record(event) — atomic O_APPEND write to the tenant's ledger.
    Caller passes optional explicit tenant_id; default uses tenantId().
  * rollup({tenant_id?, from?, to?}) — read + aggregate by model /
    tool / device. Per-tenant scoped; passes forTenant to tenantPath
    so cross-tenant queries route correctly regardless of the current
    cached tenant.
  * Helpers: ledgerExists(), ledgerSizeBytes(), __ledgerPathForTests().
- src/tenant.ts: extended tenantPath() with optional 4th parameter
  forTenant. Cross-tenant readers (cost-accounting rollup, future audit
  UI) pass it explicitly to route to the right tenant directory without
  invalidating their own cached id. Backward compatible — default keeps
  using tenantId() when omitted.
- tests/cost-accounting.test.ts — 8 unit tests covering record/rollup,
  per-tenant isolation, time-range filter, by_device fanout, missing
  ledger graceful-zero, explicit tenant_id override, corrupt-line skip.
  All 8 pass. The existing 14 tenant tests stay green.

What this PR does NOT change:
- No callers emit usage events yet. voice-agent + bodhi-realtime-agent
  + skill execution all have the data to record, but wiring them is the
  follow-up PR (one call site at a time, easier to review separately).
- No UI / report generator. `rollup()` is the contract the future
  Audit UI (§5 Next) consumes.

Test plan:
- [x] npx tsx --test tests/cost-accounting.test.ts — 8/8 pass.
- [x] npx tsx --test tests/tenant.test.ts — 14/14 pass (regression
      check after the tenantPath forTenant parameter addition).
- [x] npx tsc --noEmit clean.

Depends on PR #748 (tenant scaffold). Targets feat/tenant-identity-
scaffold as base.

Roadmap impact:
- §5 Next "Cost accounting": contract + ledger landed. Emission +
  reports are follow-ups.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ale TODO

Three fixups per sonichi's #749 review:

1. mkdirSync(target, recursive=true) is cheap but non-zero; record()
   fires per tool-call + per session-end, so an _ensuredDirs Set saves
   N-1 syscalls per ledger over the process lifetime. Pure perf, no
   correctness risk.

2. Add 'ms' to by_model. Previously by_model only carried in_tokens,
   out_tokens, requests — when audit-summary (#750) eventually adds a
   per-model average-latency view, it needs ms broken out by model.
   Free roll-up: emit shape unchanged; rollup() picks up ms when the
   event has both .model and unit='ms'. New test verifies.

3. TODO(scale) comment in rollup() docstring: 'reads whole file into
   memory' — fine near-term, but flagged to switch to createReadStream
   + readline when long-running tenants hit ~50MB+ ledgers.

Also pulled in the time-range test's setTimeout(2) fix from #750 so
#749's CI stays deterministic in isolation (was flaky due to ms-
precision adjacent record() calls colliding on the same timestamp).

Tests: 9/9 pass (was 8 — added 'by_model now breaks out ms' test).
tsc clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@qingyun-wu
qingyun-wu force-pushed the feat/cost-accounting-scaffold branch from 90fcf16 to 8a36cab Compare May 16, 2026 21:06
qingyun-wu added a commit that referenced this pull request May 16, 2026
…dger

Second consumer of PR #749's per-tenant cost ledger. Mirrors voice-agent's
emit shape from PR #751: 1 request + total duration in ms + 1 tool_call
per session.toolCalls entry. Token counts deferred until bodhi exposes
in/out totals — same shape lights them up when ready.

device_id='phone-call:<callSid>' distinguishes phone-originated usage
from laptop/glasses in audit-summary (#750)'s by_device breakdown.
Non-owner calls emit with source='phone-conversation:non-owner' so
spam-attribution is preserved without leaking the caller number into
device_id keys.

Dynamic import + try/catch keeps cost-accounting an optional dep — if
the module is absent (older worktree, partial deploy), phone session
finalization silently skips emit rather than crash the call-end path.
Mirrors #751's defensive shape.

Test plan:
- [x] npx tsc --noEmit clean.
- [ ] Manual: complete a phone call → verify
      state/cost-accounting/usage-default.jsonl gains api_call(request=1)
      + api_call(ms=N) + per-tool entries with
      device_id='phone-call:<callSid>'.
- [ ] Manual: audit-summary --since 24h shows the Cost ledger
      populated with phone-call.<callSid> in by_device.

Depends on PR #749 (cost-accounting). Targets feat/cost-accounting-
scaffold as base. Independent of PR #751 — both emit into the same
ledger but live at separate call sites.

Roadmap impact:
- §5 Next 'Cost accounting' second consumer landed. Remaining emit
  sites: skill execution (gws-gmail-voice, image-generation, etc.) and
  the Mentra bridge (PR #740).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request May 16, 2026
Activates PR #749's per-tenant cost-accounting ledger from voice-agent's
session-end path. Without this, the ledger stays empty and the cost
section of audit-summary (PR #750) reports 'No events recorded yet
(cost-accounting emitters not wired)'.

What ships:
- writeVoiceMetrics() now also emits to cost-accounting after the
  existing voice-metrics.jsonl write. Records:
  * 1 'request' event tagged with the voice model (gemini-2.5-flash
    today).
  * Total session duration in ms.
  * 1 'tool_call' per voiceToolCalls entry, with the tool name.
- Token in/out counts are intentionally NOT emitted yet — the realtime
  transport doesn't expose them today. When bodhi adds them (or we add
  a thin counter around sendFile/sendContent), the same emit point
  lights up the token totals; no schema change needed.
- Dynamic import of './cost-accounting.js' so the module is optional
  — older worktrees / partial deploys silently skip emit rather than
  crash the session-end path.

Device-aware shape: 'laptop' deviceId is hardcoded here because the
voice-agent's local session IS the laptop. Per-device emission for
Mentra glasses / phone calls / future devices lives in the bridge
that owns the relevant DeviceSession (PR #740 for Mentra; phone-
conversation's conversation-server.ts is the next consumer).

Test plan:
- [x] npx tsc --noEmit clean.
- [ ] Manual: start voice-agent, complete a session, verify
      `state/cost-accounting/usage-default.jsonl` exists with
      api_call(request=1, ms=N) + per-tool entries.
- [ ] Manual: `tsx src/audit-summary.ts --since 24h` shows the
      Cost ledger section populated.

Depends on PR #749 (cost-accounting scaffold). Targets feat/cost-
accounting-scaffold as base.

Roadmap impact:
- §5 Next 'Cost accounting' consumer landed for voice. Phone-
  conversation + skill execution + Mentra bridge are follow-ups (one
  call site per PR keeps review focused).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request May 16, 2026
…profile (#751 nit)

Per sonichi's #751 review: 3 record() calls per session-end raised a
question about fsync-per-write cost. Reality (per #749's
implementation): record() does open+write+close with no explicit fsync
— kernel batches via the page cache, so syscalls amortize cleanly.
The _ensuredDirs cache added in #749's fixup commit further reduces
the mkdir cost.

Doc-comment explains the cost profile + flags the revisit trigger
(hundreds-of-events-per-session, e.g. when bodhi exposes per-utterance
token counts) so the next reader doesn't re-debate batching at the
current scale.

No code change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request May 16, 2026
… (§5 Next)

Implements §5 Next "Audit UI built on existing audit.jsonl files. 'What
did the agent do, on which device, with which methodology, why?' A trust
feature for productization, not just a debugging artifact."

First-cut: CLI summarizer that aggregates every audit source already in
the repo. Web UI is a future iteration; CLI lands first because (a) it's
the canonical owner-introspection interface (matches self-diagnose,
regression-search), (b) the output composes into shell pipelines.

Sources consumed (any subset; missing files skipped):
- data/voice-metrics.jsonl       — voice-agent sessions
- data/call-metrics.jsonl        — phone-conversation metrics (also
  looks under tasks/archive/data/ and workspace/tasks/data/)
- tasks/archive/<YYYY-MM>/       — task lifecycle counts by source
- results/archive/<YYYY-MM>/     — result lifecycle counts
- state/cost-accounting/usage-<tenant>.jsonl  — from PR #749

Output: markdown by default, --json for machine-readable.

Usage:
  tsx src/audit-summary.ts                  # today
  tsx src/audit-summary.ts --since 7d       # last 7 days
  tsx src/audit-summary.ts --since 2026-05-01 --until 2026-05-15
  tsx src/audit-summary.ts --tenant acme    # cost rollup scoped
  tsx src/audit-summary.ts --json

What ships:
- src/audit-summary.ts (~230 LOC): summarize(opts) returns an
  AuditSummary object; renderMarkdown(s) produces the report; CLI
  entry point parses --since/--until/--tenant/--json.
- tests/audit-summary.test.ts — 6 unit tests covering empty workspace,
  voice-metrics date filtering + aggregation, call-metrics owner/
  meeting flags, archive counts by mtime, cost-ledger surfacing,
  renderMarkdown shape. All pass.
- tests/cost-accounting.test.ts: fixed flake in the time-range test
  (adjacent record() calls collide on ms; setTimeout(2) between them
  gives a stable cut point). Now deterministic.

Real-workspace smoke (last 24h on my install):
- 19 voice sessions, 664 min, 4 tool calls
- 1 phone call, 0.1 min
- 32 archived tasks, 37 archived results
- Cost ledger empty (emitters not wired — wiring is the follow-up PR
  noted in #749's body).

What this PR does NOT change:
- No web UI. CLI markdown is the first cut.
- No structured emit to audit.jsonl. The summarizer reads existing
  scattered sources; a unified audit.jsonl is a future refactor.
- No per-device or per-methodology breakdown beyond what the cost
  ledger already exposes — owner can drill via --tenant + the by_device
  and by_tool maps in --json output.

Test plan:
- [x] npx tsx --test tests/audit-summary.test.ts tests/cost-accounting.test.ts tests/tenant.test.ts — 28/28 pass.
- [x] npx tsc --noEmit clean.
- [x] Real-workspace smoke: tsx src/audit-summary.ts --since 24h
      outputs structured markdown.

Depends on PR #749 (cost-accounting) for the rollup() consumer. Targets
feat/cost-accounting-scaffold as base.

Roadmap impact:
- §5 Next "Audit UI": CLI first cut shipped. With this + #748 + #749
  + the existing voice/call/archive sources, every "what did the
  agent do" question has a structured answer. Web UI is the next iter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@qingyun-wu
qingyun-wu marked this pull request as draft May 17, 2026 05:34
… (§5 Next) (#750)

Implements §5 Next "Audit UI built on existing audit.jsonl files. 'What
did the agent do, on which device, with which methodology, why?' A trust
feature for productization, not just a debugging artifact."

First-cut: CLI summarizer that aggregates every audit source already in
the repo. Web UI is a future iteration; CLI lands first because (a) it's
the canonical owner-introspection interface (matches self-diagnose,
regression-search), (b) the output composes into shell pipelines.

Sources consumed (any subset; missing files skipped):
- data/voice-metrics.jsonl       — voice-agent sessions
- data/call-metrics.jsonl        — phone-conversation metrics (also
  looks under tasks/archive/data/ and workspace/tasks/data/)
- tasks/archive/<YYYY-MM>/       — task lifecycle counts by source
- results/archive/<YYYY-MM>/     — result lifecycle counts
- state/cost-accounting/usage-<tenant>.jsonl  — from PR #749

Output: markdown by default, --json for machine-readable.

Usage:
  tsx src/audit-summary.ts                  # today
  tsx src/audit-summary.ts --since 7d       # last 7 days
  tsx src/audit-summary.ts --since 2026-05-01 --until 2026-05-15
  tsx src/audit-summary.ts --tenant acme    # cost rollup scoped
  tsx src/audit-summary.ts --json

What ships:
- src/audit-summary.ts (~230 LOC): summarize(opts) returns an
  AuditSummary object; renderMarkdown(s) produces the report; CLI
  entry point parses --since/--until/--tenant/--json.
- tests/audit-summary.test.ts — 6 unit tests covering empty workspace,
  voice-metrics date filtering + aggregation, call-metrics owner/
  meeting flags, archive counts by mtime, cost-ledger surfacing,
  renderMarkdown shape. All pass.
- tests/cost-accounting.test.ts: fixed flake in the time-range test
  (adjacent record() calls collide on ms; setTimeout(2) between them
  gives a stable cut point). Now deterministic.

Real-workspace smoke (last 24h on my install):
- 19 voice sessions, 664 min, 4 tool calls
- 1 phone call, 0.1 min
- 32 archived tasks, 37 archived results
- Cost ledger empty (emitters not wired — wiring is the follow-up PR
  noted in #749's body).

What this PR does NOT change:
- No web UI. CLI markdown is the first cut.
- No structured emit to audit.jsonl. The summarizer reads existing
  scattered sources; a unified audit.jsonl is a future refactor.
- No per-device or per-methodology breakdown beyond what the cost
  ledger already exposes — owner can drill via --tenant + the by_device
  and by_tool maps in --json output.

Test plan:
- [x] npx tsx --test tests/audit-summary.test.ts tests/cost-accounting.test.ts tests/tenant.test.ts — 28/28 pass.
- [x] npx tsc --noEmit clean.
- [x] Real-workspace smoke: tsx src/audit-summary.ts --since 24h
      outputs structured markdown.

Depends on PR #749 (cost-accounting) for the rollup() consumer. Targets
feat/cost-accounting-scaffold as base.

Roadmap impact:
- §5 Next "Audit UI": CLI first cut shipped. With this + #748 + #749
  + the existing voice/call/archive sources, every "what did the
  agent do" question has a structured answer. Web UI is the next iter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@vidhuUC vidhuUC left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE

Solid cost-accounting implementation. The O_APPEND atomic write (PIPE_BUF) pattern is correct for JSONL — matches learn-collector discipline. The audit-summary CLI as first-cut UI is the right call before a web UI. Key things verified:

  • rollup() is the clean contract — correct to leave the audit UI as a follow-up.
  • forTenant 4th-param extension to tenantPath is backward-compatible.
  • Dynamic import + try/catch for cost-accounting in callers is the right defensive shape.
  • 8 tests covering record/rollup/ledger helpers.

The audit-summary CLI output (voice sessions, call metrics, cost rollup, task counts) is exactly what an owner needs for 'what did the agent do and what did it cost.' --json flag for machine-readable output composes well.

Depends on #748 — merge #748 first.

@qingyun-wu qingyun-wu closed this May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants