[BUG/PERF] Severe Memory Leak and Disk Swell leading to System Kernel Panic (macOS) #12687

Activity

1. dzianisv

   added

   bugSomething isn't workingSomething isn't working

   on Feb 8, 2026
2. github-actions

   assigned

   rekram1-node

   on Feb 8, 2026
3. github-actions

   added

   perfIndicates a performance issue or need for optimizationIndicates a performance issue or need for optimization

   on Feb 8, 2026
4. github-actions commented on Feb 8, 2026

   

   github-actionsbot

   on Feb 8, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue might be a duplicate of existing issues. Please check:

   • [Performance] Critical CPU usage and memory leak even when starting a new session #12513: [Performance] Critical CPU usage and memory leak even when starting a new session (similar resource exhaustion and memory leak symptoms)
   • High Memory Usage on macOS 15.7 (Apple M4, 16GB, arm64) #9239: High Memory Usage on macOS 15.7 (Apple M4, 16GB, arm64) (macOS-specific high memory usage)
   • Slow down the PC after hours usage and occupy large memory #12499: Slow down the PC after hours usage and occupy large memory (memory and performance degradation after extended use)
   • [Bug] Memory Leak: OpenCode Process Killed by 00M Killer During Extended Runtime #9743: [Bug] Memory Leak: OpenCode Process Killed by OOM Killer During Extended Runtime (similar OOM symptoms and memory exhaustion)

   Feel free to ignore if none of these address your specific case.
5. dzianisv commented on Feb 11, 2026

   

   dzianisv

   on Feb 11, 2026

   Author

   More actions

   Forensic Analysis: Kernel Panic Root Cause (Feb 11, 2026)

   Machine: MacBook Pro M1 (MacBookPro17,1), 16 GB RAM, macOS 15.5 (24F74), ~30 GB free disk

   ---

   Kernel Panic Details

   Two kernel panics in 3 days, both identical cause:

   Feb 11, 01:47 AM:

   panic(cpu 5): watchdog timeout: no checkins from watchdogd in 91 seconds
   Compressor Info: 54% of compressed pages limit (OK) and 70% of segments limit (OK)
                    with 18 swapfiles and LOW swap space
   

   Feb 9, 00:28 AM:

   panic(cpu 0): watchdog timeout: no checkins from watchdogd in 91 seconds
   Compressor Info: 52% of compressed pages limit (OK) and 95% of segments limit (OK)
                    with 23 swapfiles and LOW swap space
   

   Both times the system was so memory-starved that the kernel watchdog daemon couldn't check in within 91 seconds, triggering a hard panic and forced reboot. Boot faults recorded as wdog,reset_in_1 timeout.

   Jetsam (OOM) Events Confirm opencode as Primary Consumer

   The macOS Jetsam event from Feb 10 explicitly identifies:

   "largestProcess": "opencode"

   Memory snapshot at time of Jetsam kill:

   Process	Resident Memory	PID
   opencode	1,525.9 MB	10748
   opencode	1,350.3 MB	38850
   opencode	1,325.4 MB	78124
   opencode	1,211.9 MB	8579
   opencode	1,103.5 MB	46921
   opencode	789.1 MB	11105
   opencode	731.6 MB	60304
   opencode	465.1 MB	35103
   opencode	447.3 MB	26047
   opencode	422.6 MB	59824
   Total (opencode only)	~9.4 GB	10 instances
   node (2 instances)	~1.1 GB	—

   10 opencode processes consuming ~9.4 GB on a 16 GB machine. That's 59% of total physical RAM from opencode alone.

   System memory state at time of Jetsam:

   • Free pages: 13,781 (~215 MB) — critically low
   • Active: 221,716 pages (~3.4 GB)
   • Inactive: 210,697 pages (~3.3 GB)
   • Wired: 129,341 pages (~2.0 GB)
   • Compressor size: 437,326 pages (~6.8 GB)

   Evidence of Memory Leak

   The opencode diagnostic report from Feb 9 (opencode_2026-02-09-134500) shows clear unbounded growth:

   Footprint: 243.58 MB -> 1493.97 MB (+1250.39 MB) in ~35 minutes
   Writes: 2147.49 MB of file backed memory dirtied over 7760 seconds
            (276.75 KB/s average, exceeding limit of 24.86 KB/s)
   

   A single opencode instance grew from 244 MB to 1.5 GB in 35 minutes while also writing 2.1 GB to disk. This is characteristic of a memory leak — possibly conversation context accumulation, uncollected tool output buffers, or session state that is never freed.

   Reboot Frequency

   From last reboot, the majority of reboots are not preceded by a graceful shutdown, indicating forced restarts:

   Date	Type
   Feb 11 01:47	Kernel panic (confirmed)
   Feb 9 00:28	Kernel panic (confirmed)
   Feb 5 20:51	Graceful shutdown/reboot
   Feb 3 17:50	Graceful shutdown/reboot
   Feb 3 10:15	Forced reboot (no shutdown)
   Feb 3 01:31	Forced reboot (no shutdown)
   Feb 2 00:47	Forced reboot (no shutdown)
   Jan 31 22:41	Forced reboot (no shutdown)
   Jan 29 17:26	Forced reboot (no shutdown)
   Jan 28 14:36	Forced reboot (no shutdown)
   Jan 28 01:19	Forced reboot (no shutdown)

   ~10 forced reboots in 2 weeks. The overnight pattern (01:47, 00:28, 01:31, 00:47, 01:19) suggests opencode sessions left running overnight accumulate memory until the system panics.

   Additional Jetsam Events (Feb 7)

   Two Jetsam OOM events on Feb 7 show 5+ opencode instances ranging from 533 MB to 1,057 MB each, alongside a Python process at 6.4 GB. opencode was consistently in the top consumers.

   Summary

   1. Root cause: opencode has a memory leak — individual instances grow unboundedly (~1.25 GB in 35 min observed)
   2. Compounding factor: Stale/idle opencode sessions are not cleaned up, leading to 10+ zombie-like instances accumulating
   3. System impact: On 16 GB machines, 3–4 forgotten sessions are sufficient to exhaust RAM + swap and trigger a kernel panic
   4. Disk swell: Consistent with the 18–23 swapfiles observed; macOS creates ~2 GB swap files, so 23 × 2 GB = ~46 GB matches the ~50 GB disk swell reported in this issue
   5. Frequency: ~10 forced reboots in 2 weeks, primarily overnight when sessions are idle but still consuming memory
6. dzianisv commented on Feb 11, 2026

   

   dzianisv

   on Feb 11, 2026

   Author

   More actions

   Analysis of Related Open PRs

   There are multiple open PRs addressing the same underlying memory leak problems described in this issue, though none directly reference #12687.

   Most Relevant Open PRs

   PR	Author	Approach	Key Fixes
   #12053	@sgInnora	Targeted fixes for 6 specific leaks	AsyncQueue resolver cleanup, bash output ring buffer (10MB cap), LSP diagnostics FIFO eviction (5K files), bus subscription cleanup, PTY buffer ring buffer, process exit disposal
   #10392	@ztripez	Broad session handling fixes (8 fixes)	Provider LRU caches, remove deep clone in prompt loop, session/FileTime/ACP cleanup, compaction output deletion, pending request timeouts, processor map cleanup, string concat optimization
   #12259	@Stranmor	TTL + LRU eviction framework	ACP session TTL (1h) + LRU (50 limit), Instance cache TTL (30min) + LRU (10), LSP client limit (20), PTY RingBuffer
   #9693	@kov	O(n²) bash output fix	Streams large output to disk (>50KB threshold) instead of output += chunk string concatenation
   #9146	@sauerdaniel	ACP session lifecycle	Proper session cleanup on process exit, event listener removal
   #9147	@sauerdaniel	TUI/Slack listener leaks	Event listener cleanup
   #7914	@hendem	Subscription disposal	Dispose functions for subscription memory leaks

   Key Observations

   1. None of these PRs have been merged — all are still open with no reviews from maintainers.
   2. Significant overlap between PRs fix(memory): resolve 6 memory leak issues in long-running sessions #12053, fix(opencode): memory leaks in session handling #10392, and fix(memory): add TTL, LRU eviction, and ring buffers to prevent memory leaks #12259. They all target the same core issues (bash output accumulation, PTY buffers, session cleanup) with slightly different implementations.
   3. PR fix(opencode): memory leaks in session handling #10392 is the most comprehensive (15 files, 8 fixes) and has been tested by a user (@Thejuampi) who reported it didn't fully solve the issue for Java/jdtls LSP projects.
   4. PR fix: Prevent O(n²) memory usage in command output #9693 takes the most practical approach for the bash tool specifically — streaming to disk instead of memory accumulation.

   Root Causes Identified Across PRs

   The community has converged on these root causes:

   • Unbounded output += chunk string concatenation in bash/PTY (O(n²) memory)
   • No eviction on LSP diagnostics, provider caches, ACP sessions, or instance caches
   • Subscription/event listener leaks not disposed on session end
   • Prompt loop deep clones copying entire message history every turn
   • Compaction not actually deleting tool output from storage

   A coordinated effort to review and merge the best parts of these PRs would likely address the kernel panic symptoms reported here.
7. dzianisv commented on Feb 11, 2026

   

   dzianisv

   on Feb 11, 2026

   Last edited by dzianisv

   Author

   More actions

   Based on the forensic analysis in the issue and the PRs I reviewed:

   What the PRs would fix
   The output += chunk string concatenation issue (PRs #12053, #9693, #10392) is almost certainly a real and significant leak. This O(n²) pattern in bash/PTY output is a well-known pathological case in JavaScript/Bun — it creates massive GC pressure and intermediate string allocations that can easily explain the 244 MB to 1.5 GB growth in 35 minutes that @dzianisv observed. Multiple PRs converge on this fix independently, which gives confidence it's a real problem.
   The subscription/listener leaks (PRs #7914, #9146, #9147) are also legitimate — unremoved listeners accumulate closures that retain references to large objects.
   What the PRs likely don't fully solve
   2. @Thejuampi already tested PR #10392 (the most comprehensive one) and reported it didn't help for their Java/jdtls project. This suggests the LSP layer itself may be a major contributor that isn't adequately addressed — jdtls is notoriously memory-hungry, and simply capping diagnostics at 5K files may not be enough if the LSP client holds references to response objects, document buffers, or AST data.
   3. Overnight idle accumulation — The kernel panics happen at 01:47, 00:28, 01:31 etc. The PRs add eviction on access (LRU) and creation (FIFO), but they don't add periodic idle cleanup that would reclaim memory when opencode is sitting unused overnight. PR #12259 has a TTL-based cleanup interval for ACP sessions, but not for the other caches.
   4. No actual memory profiling — None of the PRs include heap snapshots or profiling data showing before/after. They're based on code review and reasonable assumptions, not measurement. The real dominant leak source could be something none of them address.
   Bottom line
   Merging the bash/PTY output fix (especially #9693's disk-streaming approach) and the subscription cleanup would meaningfully reduce memory growth. But I'd estimate they address maybe 40-60% of the problem. The multi-instance accumulation, LSP memory, and idle session cleanup gaps remain unaddressed. A proper fix likely requires heap profiling to identify what's actually holding memory in a real long-running session.

   👍React with 👍2Reacted by Juan Pablo Lescano and Tikiti
8. dzianisv commented on Feb 11, 2026

   

   dzianisv

   on Feb 11, 2026

   Author

   More actions

   Root Cause Analysis and Proposed Fix

   After a deep dive into the codebase, here are the actual root causes ranked by impact, and a concrete fix plan.

   1. No graceful shutdown on signals (THE #1 cause of zombie processes)

   This is the biggest problem and explains the 10 zombie processes in the forensic analysis. index.ts:153 has a finally block that calls process.exit() but never calls Instance.disposeAll(). There are zero SIGTERM/SIGINT/SIGHUP handlers anywhere in the codebase. When you close a terminal:

   • The shell sends SIGHUP to opencode
   • Bun's default handler exits immediately
   • No cleanup runs — no LSP shutdown, no MCP close, no bash kill
   • Bash tool processes are spawned with detached: true (bash.ts:168), so they survive the parent dying
   • LSP servers (up to 25 per instance) may also linger since cleanup never fires

   This single gap explains both the zombie processes AND the multiplying memory. Each time you open a terminal, run opencode, and close the terminal without a clean exit, you leave behind orphaned LSP/MCP/bash processes.

   Fix: Add SIGTERM/SIGINT/SIGHUP handlers that call Instance.disposeAll() before exiting. Call disposeAll() in the finally block.

   2. Unbounded output += chunk in bash tool (bash.ts:171-182)

   This is the #1 per-process memory growth cause. Every bash command accumulates its entire output as a string via output += chunk.toString(). For verbose builds or large find outputs, this creates O(n²) memory pressure from string concatenation. The output is only truncated after the command finishes — by then, memory has already spiked.

   Fix: Stream to disk above a threshold (like PR #9693 proposes), or use a Buffer[] ring buffer with a cap.

   3. Deep clone of entire message history every prompt loop iteration (prompt.ts:577)

   clone(msgs) deep-copies every message including all tool outputs on every loop turn. For a long conversation with large tool results, this temporarily doubles memory. Combined with MessageV2.stream() reloading all messages from disk each iteration (line 276), each turn has massive transient allocations.

   Fix: Replace the deep clone with shallow copies of only the parts that get mutated (the <system-reminder> wrapping), or mutate in place and undo after.

   4. ACP sessions never cleaned up (acp/session.ts:9)

   The sessions Map has no delete(), no clear(), no dispose callback. Sessions accumulate forever.

   Fix: Add a dispose callback to the Instance.state() call that clears the map.

   5. PermissionNext and Question missing disposal callbacks

   Unlike the legacy Permission module (which properly rejects pending promises on dispose), both PermissionNext (permission/next.ts:108) and Question (question/index.ts:82) register Instance.state() without a dispose callback. Pending promises and their closures leak on instance disposal.

   Fix: Add dispose callbacks that reject all pending promises, matching the pattern already used in the legacy Permission module.

   6. FileTime tracking grows unboundedly (file/time.ts:11-22)

   The read object tracks [sessionID][path] -> Date and entries are never evicted.

   Fix: Add cleanup when sessions end, or add a dispose callback.

   ---

   Proposed Implementation (~8 files)

   File	Change
   index.ts	Add SIGTERM/SIGINT/SIGHUP handlers that call disposeAll() before exit
   tool/bash.ts	Cap output accumulation with a ring buffer or disk streaming
   session/prompt.ts	Replace clone(msgs) with targeted shallow copies
   acp/session.ts	Add dispose callback to clear sessions map
   permission/next.ts	Add dispose callback to reject pending promises
   question/index.ts	Add dispose callback to reject pending promises
   file/time.ts	Add dispose callback to clear read tracking
   util/queue.ts	Add close() method to drain pending resolvers

   The signal handling fix alone would eliminate the zombie process problem (the 10-process / 9.4 GB scenario from the forensic analysis). Combined with the bash output cap and prompt clone fix, this covers the per-process memory growth. The remaining fixes are defense-in-depth for long-running sessions.

   This should address ~90% of the memory exhaustion and kernel panic symptoms reported in this issue.
9. dzianisv commented on Feb 11, 2026

   

   dzianisv

   on Feb 11, 2026

   Author

   More actions

   Implementation Plan

   Fix 1: Graceful shutdown via signal handlers (Critical — eliminates zombie processes)

   Problem: No SIGTERM/SIGINT/SIGHUP handlers exist anywhere. When a terminal is closed, SIGHUP kills the process instantly. Instance.disposeAll() never runs. Child processes spawned with detached: true (bash tool) survive indefinitely. LSP/MCP servers are orphaned.

   The only reliable cleanup path today is the TUI's graceful exit flow (ExitProvider -> shutdown RPC -> worker Instance.disposeAll()). Every other exit path (signals, serve command, errors) skips cleanup entirely.

   Changes:

   • index.ts: Add SIGTERM/SIGINT/SIGHUP handlers that call Instance.disposeAll() before exiting. Also call disposeAll() in the existing finally block before process.exit().
   • worker.ts: Add matching signal handlers in the worker thread, since the worker owns the Instance state and runs disposeAll().
   • serve.ts: Replace the unreachable cleanup code (await new Promise(() => {}) blocks forever, making lines 48-56 dead code) with a signal-based exit that properly cleans up.

   Fix 2: Bash/shell output accumulation (High — largest per-process memory spike)

   Problem: bash.ts:171 uses let output = "" with output += chunk.toString() on every data event. The full output string grows unboundedly during execution. The same pattern exists in prompt.ts shell() method (~line 1499). While not a persistent leak (freed when the function returns), for verbose commands this creates massive transient memory spikes with O(n²) GC pressure from string concatenation.

   Changes:

   • bash.ts: Accumulate output in a Buffer[] array. Cap total size (e.g., 10MB). Drop oldest chunks when exceeded. Join only at the end for the return value.
   • prompt.ts shell(): Apply the same ring buffer pattern.

   Fix 3: Deep clone of message history in prompt loop (High — doubles memory per turn)

   Problem: prompt.ts:577 calls clone(msgs) (remeda deep clone) on the entire session message history every loop iteration. This exists because lines 580-596 mutate part.text to wrap it in <system-reminder> tags. For long sessions with large tool outputs, this temporarily doubles memory.

   Changes:

   • prompt.ts: Instead of deep-cloning everything, create the reminder-wrapped text without mutating the original messages. Build a map of partID -> wrappedText and apply it during model message construction, or shallow-copy only the specific parts that need mutation.

   Fix 4: Missing dispose callbacks (High — pending promises leak on shutdown)

   Problem: PermissionNext (permission/next.ts:108) and Question (question/index.ts:82) register Instance.state() without dispose callbacks. The legacy Permission (permission/index.ts:53-83) does it correctly — it provides a dispose callback that rejects all pending promises. When an instance is disposed, pending promise closures (holding references to session data, tool contexts, etc.) are never freed.

   Changes:

   • permission/next.ts: Add dispose callback matching the pattern in permission/index.ts.
   • question/index.ts: Add dispose callback that rejects all pending question promises.

   Fix 5: ACP session cleanup (Medium)

   Problem: ACPSessionManager (acp/session.ts:9) stores sessions in a Map with no delete() or dispose() method. Sessions accumulate forever.

   Changes:

   • acp/session.ts: Add a dispose() method that clears the sessions map. Wire it into the instance lifecycle.

   Fix 6: AsyncQueue close mechanism (Medium)

   Problem: util/queue.ts has no close()/termination mechanism. The async iterator is infinite (while(true)). If consumers stop iterating, pending resolver callbacks stay in memory as unresolved promises.

   Changes:

   • util/queue.ts: Add a close() method that sets a flag, resolves/rejects all pending resolvers, and causes the async iterator to terminate.

   Fix 7: FileTime cleanup (Low-Medium)

   Problem: file/time.ts tracks [sessionID][path] -> Date for every file read, with no eviction or cleanup.

   Changes:

   • file/time.ts: Add a dispose callback. Add a clear(sessionID) method for session-level cleanup.

   ---

   Expected impact

   Fix	Addresses	% of reported symptoms
   Signal handlers	10 zombie processes, 9.4 GB combined, orphaned LSP/MCP/bash	~50-60%
   Bash output cap	244 MB → 1.5 GB in 35 min growth pattern	~15-20%
   Prompt loop clone	Memory doubling per turn in long sessions	~10%
   Dispose callbacks	Slow leak from pending promises on abandoned sessions	~5%
   ACP/Queue/FileTime	Defense-in-depth for long-running sessions	~5%

   Total estimated coverage: ~90% of reported symptoms.

   The signal handler fix alone eliminates the multi-process accumulation problem. Combined with the bash output cap, it addresses the two primary mechanisms that cause kernel panics on 16 GB machines.
10. dzianisv

    added 2 commits that reference this issue on Feb 11, 2026

    fix: resolve memory leaks and zombie processes from missing cleanup h…

    ...

    This issue will close once commit fcdd064 is merged into the 'dev' branch.

    fcdd064

    fix: resolve memory leaks and zombie processes from missing cleanup h…

    ...

    This issue will close once commit 4cb915f is merged into the 'dev' branch.

    4cb915f
11. dzianisv commented on Feb 11, 2026

    

    dzianisv

    on Feb 11, 2026

    Author

    More actions

    PR #13183 — Review and Cleanup

    After a thorough self-review of the initial commit, I found and corrected several issues:

    Problems caught and fixed

    1. serve.ts imported modules that don't exist on dev — Whisper, Coqui, Tunnel were accidentally included from a local feature branch. This would have broken the upstream build. Removed all voice/tunnel code, kept only the signal-based exit + Instance.disposeAll() fix.

    2. bash.ts metadata preview bug — chunks.slice(0, 1).toString() calls Array.toString() (joins with commas), not Buffer.concat(). This would produce garbled TUI output. Replaced with a separate preview string built incrementally and capped at 30KB.

    3. prompt.ts shell metadata called Buffer.concat() on every data chunk — This is O(n²), the same problem as the original output +=. Fixed with the same capped preview approach.

    4. Removed ACPSessionManager.dispose() — Nobody calls it. The class is instantiated in ACPAgentBridge with no lifecycle hook to the Instance state system. Cosmetic change that fixes nothing.

    5. Removed AsyncQueue.close() — Nobody calls it. The two AsyncQueue instances in tui.ts are module-level singletons with no dispose lifecycle. Cosmetic change that fixes nothing.

    Final PR: 8 files, all meaningful

    Fix	Files	What it prevents
    Signal handlers + Instance.disposeAll()	index.ts, worker.ts, serve.ts	Zombie processes — the primary cause of #12687
    Buffer[] ring buffer + capped preview	bash.ts, prompt.ts	OOM from O(n²) string growth on verbose commands
    Shallow copy vs deep clone	prompt.ts	Memory spikes from clone(msgs) every prompt loop
    Dispose callbacks	permission/next.ts, question/index.ts	Promise closure leaks on instance disposal
    FileTime cleanup	file/time.ts	Unbounded growth of session read records

    Testing

    • tsgo --noEmit passes across all 17 packages
    • 844 tests pass across 59 files, 0 failures
    • No behavioral changes — all fixes follow existing codebase patterns

    PR: #13183
12. 37 remaining items

    Load more

13. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Another leak-path hardening commit pushed:

    • dzianisv@c055e6baa

    Change:

    • FileTime now uses bounded maps instead of unbounded nested objects.
    • Added caps with env overrides:
      • OPENCODE_FILETIME_SESSION_MAX (default 256)
      • OPENCODE_FILETIME_FILE_MAX (default 1024)
    • Oldest entries are evicted LRU-style for both per-session file reads and total session tracking.

    Why:

    • Prevent unbounded growth of read-timestamp metadata during long-running sessions with heavy file churn.

    Validation:

    • added eviction tests in test/file/time.test.ts
    • targeted suites pass (file/time, session, acp, mcp)
    • package typecheck passes.
14. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Another focused fix landed for high-output shell workloads:

    Commits:

    • dzianisv@e3fdd9a50
    • dzianisv@476d693f1

    What changed:

    • SessionPrompt.shell now:
      • caps in-memory stdout/stderr capture (OPENCODE_SHELL_CAPTURE_MAX_BYTES, default 2MB)
      • throttles incremental message.part.updated metadata emissions
      • truncates persisted final output via existing Truncate.output(...)
    • BashTool now:
      • caps in-memory capture (OPENCODE_BASH_CAPTURE_MAX_BYTES, default 2MB)
      • throttles metadata updates
      • surfaces clipped metadata flag

    Validation:

    • targeted tests pass (tool/bash + session/acp/mcp + file/time suites)
    • typecheck passes
    • stress repro (POST /session/:id/shell with ~8MB output command, 12 times):
      • before this patch: process RSS grew to ~1915MB
      • after this patch: process RSS stayed around ~493MB
      • process count stayed flat (2) in both cases

    This removes another major memory pressure source during shell-heavy sessions.
15. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Quick follow-up on the snapshot: false workaround people reported:

    • Confirmed: snapshot is a valid top-level config key in opencode.json.
    • Runtime behavior: when set to false, OpenCode skips workspace Git snapshot tracking used for step diff/revert flows.
    • This can reduce memory/IO pressure in long-running sessions, but it is a mitigation, not the root-cause fix.

    I also pushed a docs/schema update so this option is discoverable:

    • docs section added: packages/web/src/content/docs/config.mdx (### Snapshots)
    • schema description added: packages/opencode/src/config/config.ts
    • commit: dzianisv@267433556

    This should address the "option exists but not documented" confusion while leak root-cause work continues.
16. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Consolidated update: applied fixes + practical memory mitigations

    Fixes already implemented (with commits)

    1. Lazy MCP bootstrapping for status/bootstrap paths (avoid process fanout)

    • commit: dzianisv@7c7fc91d4

    2. Dynamic MCP command refresh + lazy-connect coverage tests

    • commit: dzianisv@89ecf7b67

    3. MCP tool runtime opt-in (no implicit auto-connect fanout)

    • commit: dzianisv@84a1a0aaa

    4. Replace deep clone churn on part update publish path

    • commit: dzianisv@324d93459

    5. Bound FileTime read state (session/file map caps)

    • commit: dzianisv@c055e6baa

    6. Cap + throttle shell output streaming updates (/session/.../shell + bash tool)

    • commit: dzianisv@e3fdd9a50

    7. Docs for shell memory guardrails

    • commit: dzianisv@476d693f1

    8. Config/docs visibility for snapshot: false workaround

    • commit: dzianisv@267433556

    Workarounds / options to reduce memory usage now

    1. Disable workspace snapshots (can reduce memory + I/O in long sessions):

    {
      "snapshot": false
    }

    2. Keep MCP servers disabled unless needed (mcp.<name>.enabled: false), then enable selectively.

    3. Prefer lower-output shell usage patterns (very large continuous stdout/stderr still creates pressure even with caps/throttling).

    4. Keep compaction enabled (compaction.auto: true, compaction.prune: true).

    Important note

    snapshot: false is a mitigation and can help significantly for some workloads, but it is not a full root-cause guarantee by itself. The larger stability improvement comes from combining it with the runtime fixes above (MCP fanout prevention + shell stream caps/throttling + reduced clone churn + bounded file-time state).

    Current status

    After these changes, stress probes showed major RSS reduction on high-output shell scenarios versus pre-fix runs. Continuing to monitor long multi-session usage for remaining growth vectors (especially long-lived session/message retention patterns and LSP-heavy workloads).
17. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Update: I rebased/squashed memory-fix history on dzianisv/opencode:dev into commit-per-fix units. Canonical commits are now:

    1. c673d81df — serve memory workload profiler (+ threshold stop/docs consolidated)
    2. f6ebb6f8c — share + hard-close MCP clients across instances
    3. ed4756748 — dedupe instance bootstraps + cache headroom tuning
    4. 3cd75c562 — MCP lazy/opt-in runtime path + command refresh/tests + docs consolidation
    5. 3035ace23 — remove deep clone churn on part update bus events
    6. 53afa4e3c — cap file read timestamps per session (bounded growth)
    7. b0e7dbcc8 — cap + throttle shell output streaming (docs consolidated)
    8. 8dfa12ac5 — document snapshot: false workaround in schema/docs

    No functional delta from the prior tip commit; this was history consolidation for cleaner review and traceability.
18. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Additional fix shipped for shell-tool memory safety:

    • fix(bash): spool output to disk instead of in-memory capping
    • commit: dzianisv@9b16b0c33

    What changed:

    1. Bash tool no longer relies on an in-memory hard cap for captured output.
    2. Stdout/stderr are streamed to a file under OpenCode tool-output storage using backpressure-aware writes.
    3. In-memory state keeps only a bounded preview for metadata/UI updates.
    4. Tool response is still truncated for model context limits, but now references the full persisted output path.

    This keeps memory bounded while preserving full command output for post-processing (Read/Grep) without clipping the underlying output stream.
19. OpenCodeEngineer commented on Mar 15, 2026

    

    OpenCodeEngineer

    on Mar 15, 2026

    More actions

    Traceability update: added a commit index for memory-related fixes in fork README.

    • commit: dzianisv@9fbc40783
    • file: README.md (Fork Memory Optimizations (dzianisv/dev) section)

    This section links the critical memory optimization commits/profiling additions in one place for easier review and bisecting.
20. dzianisv commented on Mar 16, 2026

    

    dzianisv

    on Mar 16, 2026

    Author

    More actions

    Findings from the memory investigation:

    This was not caused by SQLite mmap growth or a 512 MB Bun heap cap.

    What the code actually does:

    • serve starts the memory monitor via Memory.start("serve"), but that is diagnostics only, not a heap limiter.
    • The relevant flags are OPENCODE_MEMORY_MONITOR*.
    • The docs example uses OPENCODE_MEMORY_MONITOR_THRESHOLD_MB=5120, which is a snapshot/monitor threshold, not a hard cap.
    • The frontend debug bar also shows performance.memory.jsHeapSizeLimit, but that is Chromium-only client-side memory, not the Bun server.

    What caused the spike instead:

    • The sidebar/session warm-prefetch path was requesting full GET /session/:id/message?limit=200 payloads.
    • Some persisted sessions contained very large tool and reasoning parts.
    • The Bun server had to hydrate those rows into JS objects/strings and JSON-encode them, which drove WebKit Malloc / Bun JSC heap into the multi-GB range.

    Evidence:

    • Live DB had about 458 MB of tool parts and about 92.8 MB of reasoning parts, with two sessions around 183 MB each.
    • SQLite mmap_size was 0, cache size was small, and DB file size was about 624 MB.
    • vmmap growth was concentrated in WebKit Malloc, not mapped SQLite regions.

    Fix validated locally:

    • Added preview=true mode for session message fetches so sidebar prefetch only loads lightweight preview payloads.
    • Synthetic route check with a 10 MB persisted tool output:
      • full response: 10,001,262 bytes
      • preview response: 895 bytes
      • reduction: 99.99%
21. dzianisv commented on Mar 16, 2026

    

    dzianisv

    on Mar 16, 2026

    Author

    More actions

    I hit a concrete live serve failure that appears to be part of this broader memory/perf issue.

    Observed on a long-running opencode serve instance on macOS:

    • the serve process stopped responding on every route, including /global/health
    • just before the wedge, dev.log recorded database or disk is full from Session.updatePart()
    • the host disk itself was not full (df still showed ~17 GiB free)
    • sample/watchdog data showed the old Bun process peaking around 7.9 GiB physical footprint and ~5.5 GiB process-tree RSS
    • script/memory-watchdog.sh also started failing with printf: write error: No space left on device

    The strongest code-path correlation I found is large-session history hydration on the prompt hot path:

    • SessionCompaction.prune() calls Session.messages({ sessionID })
    • SessionSummary.summarize() also calls Session.messages({ sessionID })
    • Session.messages() fully hydrates the session with part payloads
    • prune was also being fired without awaiting, so repeated turns can stack background work

    That combination is a plausible path to:

    • memory blow-up on large sessions
    • transient ENOSPC / swap pressure on macOS
    • SQLITE_FULL while writing part updates
    • wedged serve process that stops answering even health checks

    I patched this locally in a working tree to reduce hot-path pressure:

    • changed compaction prune to page backward through message history instead of loading the full session
    • added per-session scheduling/coalescing for prune
    • added per-session scheduling/coalescing for session summary work
    • added regression coverage for prune behavior

    Focused validation on the local patch:

    • bun test test/session/compaction.test.ts test/session/session.test.ts passed
    • bun typecheck passed

    This is not the same as actual disk-full conditions in #17589; here the machine still had free disk when SQLite started throwing database or disk is full.
22. OpenCodeEngineer commented on Mar 31, 2026

    

    OpenCodeEngineer

    on Mar 31, 2026

    More actions

    Investigation update (2026-03-31 UTC) from a live run on macOS:

    • opencode serve briefly reached >2 GiB RSS, then rotated to a new PID.
    • Current diagnostics from /global/memory?children=true show process-tree RSS in the ~1.5-2.6 GiB range with 16-23 descendants while root process RSS is typically ~475-605 MiB.
    • Dominant memory is in MCP descendants, not PTYs: duplicated chrome-devtools-mcp stacks (--autoConnect and --browserUrl), plus playwriter, whisper-mcp, @modelcontextprotocol/server-memory, and telemetry watchdog child processes.
    • Descendant tree sample: descendants=23 rss_mb=1744.6; multiple MCP processes are each in the ~120-180 MiB range.
    • dev.log shows high MCP churn: many repeated failed to get prompts errors (playwriter/whisper) and repeated shared-client reuse/close activity.
    • watchdog-stderr.log is repeatedly logging /Users/engineer/workspace/opencode/script/memory-watchdog.sh: No such file or directory (thousands of lines), so watchdog instrumentation appears misconfigured/broken in this setup.
    • Session counts are high (total=333, active=141), but pty.active=0 and instance cache is bounded (size<=4), so this does not look like a PTY leak.

    Hypothesis: the recurring memory pressure is primarily MCP process fan-out/churn (including duplicate CDP MCP entries) plus prompt-probe retry noise, not a single PTY leak path.
23. dzianisv commented on Mar 31, 2026

    

    dzianisv

    on Mar 31, 2026

    Author

    More actions

    rebooted overnight

    Primary culprit was opencode process fan-out, with Chrome as the biggest single process.

    • Crash time: March 31, 2026 at 03:08:13 -0700.
    • Reboot trigger from panic log: watchdog timeout: no checkins from watchdogd in 94 seconds with 9 swapfiles and LOW swap space in /Library/Logs/DiagnosticReports/panic-full-2026-03-31-
      030813.0002.panic.
    • At panic snapshot:

    1. opencode: 995 processes, 23.8 GB RSS total (~53.3% of all process RSS captured).
    2. Google Chrome* family: 9.1 GB RSS total (~20.4%).
    3. Largest single process: Google Chrome Helper (Renderer) PID 56594 at 4.96 GB RSS.
    4. Largest opencode process: PID 49761 at 0.88 GB RSS, plus many opencode processes clustered around ~240–260 MB.

    So if you want one process answer: opencode (runaway process count) caused most of the memory pressure; Chrome renderer was the top individual memory hog that worsened it.
24. OpenCodeEngineer commented on Mar 31, 2026

    

    OpenCodeEngineer

    on Mar 31, 2026

    More actions

    Root-cause update from panic analysis (Mar 31, 2026 03:08:13 -0700): this was not 995 "agents"; it was a large subprocess storm.

    Exact spawn path in code

    1. GET /agent route calls Agent.list()
       • packages/opencode/src/server/server.ts lines 434-453
    2. Agent.list() resolves state that calls Config.get()
       • packages/opencode/src/agent/agent.ts lines 75, 80, 322-324
    3. Config.get() loops config dirs and schedules dependency checks per dir
       • packages/opencode/src/config/config.ts lines 143-159
    4. needsInstall() (local channel) calls PackageRegistry.isOutdated("@opencode-ai/plugin", ...)
       • packages/opencode/src/config/config.ts lines 356-359
    5. PackageRegistry.info() executes:
       • Process.run([which(), "info", pkg, field], ...)
       • packages/opencode/src/bun/registry.ts line 50
       • which() resolves to process.execPath, so the command is effectively:
         • opencode info @opencode-ai/plugin version

    Evidence from panic + runtime

    • Panic processByPid had 995 entries with procname="opencode".
    • All 995 had thread name HTTP Client (signature of those helper subprocesses).
    • At runtime under opencode serve, child processes with PPID of the server include multiple concurrent:
      • /Users/engineer/.local/bin/opencode info @opencode-ai/plugin version
    • Logs during churn also show heavy repeated polling and instance churn (creating instance + No context found for instance).

    So the fan-out is coming from repeated plugin version checks (opencode info ...) triggered through the /agent -> Agent.list -> Config.get path under high polling/workspace churn.
25. OpenCodeEngineer commented on Mar 31, 2026

    

    OpenCodeEngineer

    on Mar 31, 2026

    More actions

    Upstream duplicate scan update (2026-03-31):

    I searched open/closed issues and PRs for this exact signature (opencode info @opencode-ai/plugin version subprocess fan-out from /agent polling path).

    Result:

    • I did not find a prior issue/PR that already documents this exact fan-out root cause.
    • The closest matches are broader memory/process tickets, not the same trigger path:
      • Multiple memory leaks cause unbounded RAM growth during extended TUI usage #16697 (umbrella memory leaks)
      • [Bug]: Massive memory leak (60GB+ OOM crash) on Server #17908 (serve OOM spike)
      • Opencode processes survive terminal disconnect, become orphans, and leak memory #12913 (orphan processes)
      • Memory leak: serve and web commands leak child processes on exit; SSE heartbeat interval leaks on server-initiated close #14091 (child-process leak on exit)
    • Plugin registry-related fixes exist (for example fix(core): read stdout and stderr in PackageRegistry.info before waiting for the process to exit #16998 on PackageRegistry.info output handling), but they do not address repeated concurrent info subprocess spawning under high polling/churn.

    Conclusion: this specific subprocess-storm path appears newly identified and should remain tracked here.
26. thdxr commented on Apr 2, 2026

    

    thdxr

    on Apr 2, 2026

    Member

    More actions

    Closing in favor of #20695 where we're collecting all memory-related problems. Please follow that issue for updates.
27. thdxr

    closed this as completedon Apr 2, 2026