Skip to content

refactor(conversation-store): drop triple-encoded sessions JSON + duplicate events (closes #1052) - #1054

Merged
sonichi merged 1 commit into
mainfrom
feat/conversation-store-drop-redundancy
May 23, 2026
Merged

sonichi merged 1 commit into
mainfrom
feat/conversation-store-drop-redundancy

Conversation

@liususan091219

Copy link
Copy Markdown
Collaborator

Closes #1052. Stacks on top of #1051 — the diff currently includes #1051's commits; once that merges this PR rebases to the clean redundancy-cleanup-only diff.

Why

Per the table in #1052, three storage paths encode the same atoms:

atom source 1 (canonical post-#1051) source 2 source 3
user / agent utterance voice / phone / discord_voice row (per-utt) session_events event_name='user:<text>' / 'sutando:<text>' sessions.events JSON
tool call surface row kind='tool_call' (realtime, #1051) session_events event_name='tool_call:<name>' / 'tool_result:...' sessions.tool_calls JSON

Row-count proof on the live workspace db pre-this-PR: voice 3673 rows but session_events 20027 rows (the pre-#1051 v_voice query reading 'user:'/'sutando:'/'tool_call:' prefixes returned ~10574 rows — 3× the surface table for the same data). Cost: more disk, more confused readers (you can SELECT from any of 3 places and get the same number).

What

Single source of truth per atom:

  • Utterances: surface tables voice / phone / discord_voice only.
  • Tool calls: surface tables only (kind='tool_call', written real-time via recordToolCall from refactor(conversation-store): per-surface tables + realtime tool_call writes #1051).
  • Lifecycle events (session_started, session_ended, error, task_delegated, etc.): session_events only.
  • sessions keeps just the per-session rollup metadata (duration, transcript_lines, tool_count, etc.) — no more JSON columns.

Schema change

ALTER TABLE sessions DROP COLUMN tool_calls;
ALTER TABLE sessions DROP COLUMN events;

Guarded via PRAGMA table_info so re-init is a no-op. SQLite 3.35+ required (node:sqlite ships with newer than that).

Writer-side changes

file line(s) dropped
src/conversation-store.ts recordSession tool_calls + events cols in INSERT; session_events fan-out filter for user:/caller:/sutando:/assistant:/tool_call:/tool_result: prefixes
src/voice-agent.ts onToolCall, onToolResult, turn-handler tool_call:, tool_result:, ${evtRole}:${content} pushes
skills/phone-conversation/scripts/conversation-server.ts onToolCall, onToolResult, turn-handler tool_call:, tool_result:, caller:, sutando: pushes
skills/discord-voice/scripts/discord-voice-server.ts onToolCall, onToolResult, turn-handler tool_call:, tool_result:, user:, sutando: pushes

Lifecycle pushes (session_started, session_ended, error:, task_delegated:, task_result:, channel_result:, rec_indicator:, call_ended) preserved across all 3 surfaces.

Reader-side changes

  • skills/regression-search/scripts/diagnose-call.py:_build_transcript already updated in refactor(conversation-store): per-surface tables + realtime tool_call writes #1051 to read surface tables (UNION ALL + kind != 'tool_call' filter).
  • skills/regression-search/scripts/diagnose-call.py:find_metrics updated here:
    • tool_calls list rebuilt from surface tables (UNION ALL + kind = 'tool_call' WHERE session_id = ?)
    • events list pulled directly from session_events WHERE session_id = ? OR call_sid = ?
    • Output shape ({name, durationMs, timestamp} per tool call; {event, timestamp} per event) unchanged.

SessionMetrics interface keeps toolCalls + events fields for backwards compat with existing callers — silently ignored / filtered now instead of persisted.

Verification

  • ✅ tsc --noEmit clean
  • ✅ Migration applied to live workspace db: sessions 13 → 11 columns; existing rows preserved with metadata intact
  • ✅ All 3 surface servers (voice / phone / discord-voice) restart cleanly on the new code
  • ✅ diagnose-call.py syntax check + structural rewrite

Risks

  • External callers reading sessions.tool_calls or sessions.events JSON outside this repo will break post-merge. Mitigation: anyone affected can rebuild the same data via the surface tables (for tool_calls) or session_events (for events). Migration log line prints when columns are dropped so the operator notices.
  • Historical sessions' JSON data is gone — DROP COLUMN doesn't preserve dropped column data. Migration log makes this clear. Surface tables + session_events have all the same atoms going forward; pre-refactor(conversation-store): per-surface tables + realtime tool_call writes #1051 history was already partially recoverable from those.

🤖 Generated with Claude Code

…licate events (closes #1052)

Builds on #1051. Three storage paths in conversation.sqlite were triple-
encoding the same atoms — per #1052's table. This PR collapses them so
surface tables (voice / phone / discord_voice) and session_events become
the single sources of truth; the sessions table keeps only the per-
session rollup metadata.

Schema migration (idempotent, init-time):
- ALTER TABLE sessions DROP COLUMN tool_calls (now in surface tables as
  kind='tool_call' rows, written real-time per #1051)
- ALTER TABLE sessions DROP COLUMN events (now in session_events as
  lifecycle-only rows, see below)
- Guarded via pragma_table_info — no-op on re-run.

Writer-side (3 surface servers + store):
- recordSession() INSERT drops the 2 JSON columns (13→11 cols)
- recordSession()'s session_events fan-out now skips user:/caller:/
  sutando:/assistant:/tool_call:/tool_result: prefixes (defense-in-depth
  filter for any external caller still passing them in m.events)
- voice-agent.ts removes 3 voiceEvents.push: tool_call: (onToolCall),
  tool_result: (onToolResult), and the user:/sutando: utterance loop in
  the turn-handler. logConversation() + recordToolCall() are the
  canonical writes for those.
- conversation-server.ts (phone) removes the same 4: tool_call:,
  tool_result:, caller:, sutando:
- discord-voice-server.ts removes the same 4: tool_call:, tool_result:,
  user:, sutando:
- Lifecycle pushes (session_started, session_ended, error, task_*,
  channel_result, rec_indicator, call_ended) are preserved.

Reader-side:
- diagnose-call.py (skills/regression-search) rebuilds the tool_calls
  list from surface tables (UNION ALL 3 with kind='tool_call' filter
  by session_id) and the events list from session_events directly. The
  output shape is unchanged.

SessionMetrics interface kept toolCalls + events fields for backwards
compat with existing callers — they're now silently ignored (toolCalls)
or filtered (events) instead of persisted as JSON blobs.

Verified end-to-end:
- Migration applied to live workspace db; sessions now has 11 columns;
  pre-existing data preserved.
- All 3 surface servers restart cleanly under new code.
- tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@liususan091219
liususan091219 force-pushed the feat/conversation-store-drop-redundancy branch from ff625b8 to c7b314e Compare May 23, 2026 13:43

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clean redundancy cleanup after #1051's per-surface tables landed. Per #1052's analysis:

  • Drops the triple-encoded sessions.events JSON column + duplicate session_events rows (event_name='user:<text>' / 'sutando:<text>' / 'tool_call:<name>') that the #1051 per-surface tables made redundant.
  • Single source of truth post-merge: per-surface row in voice / phone / discord_voice tables. session_events keeps only system events (SESSION_END etc.), not utterance/tool-call duplicates.
  • Now-clean diff after Maddy's rebase: 1 commit (c7b314e), +130/-49 — visible in the PR diff. Rebase resolved the stacking-on-#1051 conflict.

Per Maddy at 13:51Z. Ready for merge.

@sonichi
sonichi merged commit 9457046 into main May 23, 2026
2 checks passed
@sonichi
sonichi deleted the feat/conversation-store-drop-redundancy branch May 23, 2026 13:54
liususan091219 added a commit to liususan091219/sutando-fork that referenced this pull request May 28, 2026
…licate events (closes sonichi#1052) (sonichi#1054)

Builds on sonichi#1051. Three storage paths in conversation.sqlite were triple-
encoding the same atoms — per sonichi#1052's table. This PR collapses them so
surface tables (voice / phone / discord_voice) and session_events become
the single sources of truth; the sessions table keeps only the per-
session rollup metadata.

Schema migration (idempotent, init-time):
- ALTER TABLE sessions DROP COLUMN tool_calls (now in surface tables as
  kind='tool_call' rows, written real-time per sonichi#1051)
- ALTER TABLE sessions DROP COLUMN events (now in session_events as
  lifecycle-only rows, see below)
- Guarded via pragma_table_info — no-op on re-run.

Writer-side (3 surface servers + store):
- recordSession() INSERT drops the 2 JSON columns (13→11 cols)
- recordSession()'s session_events fan-out now skips user:/caller:/
  sutando:/assistant:/tool_call:/tool_result: prefixes (defense-in-depth
  filter for any external caller still passing them in m.events)
- voice-agent.ts removes 3 voiceEvents.push: tool_call: (onToolCall),
  tool_result: (onToolResult), and the user:/sutando: utterance loop in
  the turn-handler. logConversation() + recordToolCall() are the
  canonical writes for those.
- conversation-server.ts (phone) removes the same 4: tool_call:,
  tool_result:, caller:, sutando:
- discord-voice-server.ts removes the same 4: tool_call:, tool_result:,
  user:, sutando:
- Lifecycle pushes (session_started, session_ended, error, task_*,
  channel_result, rec_indicator, call_ended) are preserved.

Reader-side:
- diagnose-call.py (skills/regression-search) rebuilds the tool_calls
  list from surface tables (UNION ALL 3 with kind='tool_call' filter
  by session_id) and the events list from session_events directly. The
  output shape is unchanged.

SessionMetrics interface kept toolCalls + events fields for backwards
compat with existing callers — they're now silently ignored (toolCalls)
or filtered (events) instead of persisted as JSON blobs.

Verified end-to-end:
- Migration applied to live workspace db; sessions now has 11 columns;
  pre-existing data preserved.
- All 3 surface servers restart cleanly under new code.
- tsc --noEmit clean.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[conversation-store] remove triple-encoded redundancy after per-surface tables (#1051 follow-up)

2 participants