Repository navigation
Gateway keeps ~33 GB RSS after every workspace is unloaded, so the node refuses new engines #1590
Description
Problem
A Linux build node's gateway (prod-code 0.3.41, uptime about 42 h) reports workspaces 0 sessions 0 commands 0 and rss 33638 MB in prod-code cluster, with host memory at 81%. Its loaded engines are gone, but the memory they held is not returned. The capacity check therefore refuses every new engine:
Error: gateway refused the session: capacity: this node has no memory for a new rust engine (memory 81% used, 0.0 GiB held for 0 load(s) in flight, a new engine counts 4.0 GiB, the limit is 85% of 125.0 GiB). Engines already loaded here keep answering. Retry in a few minutes, once loads in flight settle or idle engines are unloaded, ...
The retry advice cannot help: nothing is in flight and nothing is loaded. A checkout placed on that node (rendezvous hashing) loses navigation (prod-code def, refs, outline) until the gateway restarts. Another node with 6 workspaces and 10 sessions shows a comparable rss (~31.8 GB), so ~33 GB with none is retention, not load.
Reproduce
Run prod-code cluster against a node whose engines were unloaded by idle eviction after heavy use. Compare rss with workspaces 0, then run a navigation command from a checkout placed there.
Expected
After its last engine is unloaded, the gateway's rss returns close to its idle baseline: the allocator releases memory, or engines run in processes that exit. Failing that, the capacity check could count memory the gateway itself retains as reclaimable, and say a restart is what frees it.
Environment
- client: prod-code 0.3.41 (commit 9a21810) on macos aarch64
- node: linux x86_64 (prod-code 0.3.41, commit 9a21810), engines: rust (ra_ap_ide), go (gopls), cpp (clangd), python (basedpyright-langserver), typescript (tsc --lsp), java (jdtls), kotlin (kotlin-language-server), csharp (omnisharp), zig (zls), scala (metals), generic-lsp
Private details: report inc-2026-10-26-prodcode-ram7-idle-rss.
Filed with prod-code report-issue.
Activity
fix(gateway): unloading engines gives their freed heap back to the sy…
alex09x commentedon Oct 28, 2026
PR #1666 was merged at bd5509e. It calls
malloc_trim(0)on Linux/glibc after dropping unloaded engines, and its focused test demonstrates freed heap pages returning to the OS. The PR records 3/3 Linux passes for that allocator test.The issue remains open because this test demonstrates allocator reclaim for a 256 MiB synthetic allocation, not that the reported ~33 GiB after unloading a real analysis engine is reclaimed on a gateway. The PR explicitly requires a post-unload memory log from a node for that field confirmation. No node restart, deployment, limit change or live configuration change was made here.
alex09x commentedon Oct 28, 2026
Interim rollout evidence: v0.3.42 is deployed to all nine gateways and both local client paths. Immediately after the normal rollout, every remote gateway was healthy with zero workspaces, sessions and running commands. The previously observed idle Linux gateway fell from about 33.36 GB RSS to about 15.8 MB after its normal restart.
This is only a restart baseline and does not prove allocator trimming after an engine unload. The issue remains open until a real Rust workspace is loaded and unloaded on a v0.3.42 Linux gateway and the
[MEMORY] freed heap returned to the system after unloadinglog line plus before/after RSS are recorded.alex09x commentedon Oct 28, 2026
Additional field measurement on v0.3.42: after 1,800 seconds idle, a real Rust workspace unloaded on a Linux gateway. The gateway logged
[MEMORY] freed heap returned to the system after unloadingwithrss_before_mb=8713andrss_after_mb=7853(about 860 MB returned).The gateway still had four other loaded workspaces and one active session at that point, so this does not establish the zero-workspace steady-state or explain the original ~33 GB retention. Immediately after the normal v0.3.42 rollout, the previously affected idle gateway was healthy at about 15.8 MB with no workspaces, sessions or commands. Keeping this issue open until a zero-workspace observation and residual RSS are accounted for.
alex09x commentedon Oct 28, 2026
Clean zero-workspace unload measurement on v0.3.42 (2026-10-28): after 1,800 seconds idle, a real Rust workspace was evicted. The gateway logged
rss_before_mb=2200 rss_after_mb=1333(about 867 MiB returned). The subsequent cluster snapshot showed zero loaded workspaces, active sessions, active queries, and running commands; RSS was 1,398,493,184 bytes (about 1.30 GiB). The same gateway had been about 19 MiB immediately after the normal restart earlier in the rollout. This confirms heap reclamation on unload but leaves a large residual RSS unexplained, so the issue remains open while retained owners are investigated.alex09x commentedon Oct 28, 2026
Second clean unload measurement on v0.3.43 (2026-10-28): after 1,800 idle seconds,
[MEMORY]loggedrss_before_mb=2375 rss_after_mb=1389. The subsequent cluster snapshot showed zero workspaces, active sessions, active queries, and running commands;/procshowed about 1.40 GiB RSS. The process retained 32 threads namedVfsLoaderafter the workspace was evicted, alongside 32 Tokio workers. Source inspection shows VFS directory loading uses the process-global Rayon pool; limiting glibc allocator arenas at gateway startup is the next field-tested fix. The issue remains open until a new-release probe shows post-unload RSS near idle baseline and a fresh workspace still loads normally.alex09x commentedon Oct 29, 2026
Post-release v0.3.44 field probe (2026-10-29): a real Rust workspace was loaded on a Linux gateway and left idle for 1,800 seconds. The gateway logged
[MEMORY] rss_before_mb=2903 rss_after_mb=565; the immediate post-eviction snapshot had zero loaded workspaces, sessions, queries, and running commands, and/procRSS was 586,736 kB. A separate fresh source snapshot then loaded successfully, andcode_definitionreturnedreturn_freed_memory. Startup logs confirmedmax_arenas=4.This is a substantial improvement over the clean v0.3.43 probe (2,375 -> 1,389 MiB), with post-unload RSS lower by 824 MiB. However, this probe did not capture v0.3.44's immediate post-restart idle baseline, so it does not yet establish the requested near-baseline result. I am reopening the issue while checking the remaining 565 MiB retained after unload.
alex09x commentedon Oct 29, 2026
Follow-up to the v0.3.44 probe: after verifying the second workspace load, I restarted the same gateway with no active sessions to measure a clean v0.3.44 idle baseline. It reported 14 MiB RSS with zero workspaces, sessions, queries, or commands. The prior clean post-eviction measurement was 565 MiB, so it is not close to this release's idle baseline even though it is 824 MiB below v0.3.43's residual. #1590 remains open while I investigate the remaining retained memory; the successful second workspace load is verified.
alex09x commentedon Oct 29, 2026
Additional bounded allocator/thread-pool experiments on v0.3.44 used a temporary 60-second idle-eviction interval on one idle Linux gateway. Each run loaded one fresh source snapshot, executed
code_definition, waited for eviction, and checked the[MEMORY]log. Results (post-unload RSS):MALLOC_ARENA_MAX=1with default tcache: 1,422 MiB.tcache_count=0, default Rayon pool: 1,327 MiB.RAYON_NUM_THREADS=4alone: 1,369 MiB.tcache_count=0+ Rayon 4: 333 MiB (best observed); the separate fresh-workspace reload had succeeded.tcache_count=0+ Rayon 2: 1,301 MiB.tcache_count=0+ Rayon 4: 1,287 MiB.tcache_count=0+ Rayon 1: 1,276 MiB.The settings were experimental only and have all been removed. The gateway is back on the published v0.3.44 defaults, healthy with zero workspaces/sessions/commands and a 14 MiB clean-start RSS. These short-timeout results do not prove a durable improvement; none matched the clean-start baseline, so the issue remains open for a code-level root-cause fix.
alex09x commentedon Oct 29, 2026
Follow-up repeatability check on the v0.3.44 default allocator (only the eviction timer was temporarily shortened to 60 seconds): three consecutive fresh Rust workspace loads each passed
code_definition; post-unload[MEMORY]results were 3,239 -> 297 MiB, 2,346 -> 1,216 MiB, and 3,998 -> 294 MiB. Every post-eviction snapshot showed zero workspaces, sessions, queries, and running commands. The residual RSS is non-monotonic rather than steadily growing, but varies substantially. I removed the temporary timer and restarted the gateway; it is back to the v0.3.44 defaults with 15 MiB clean-start RSS. The issue remains open pending a decision on whether stable warm-idle behavior or proximity to cold-start RSS is the acceptance target.alex09x commentedon Oct 29, 2026
Resolution for the original #1590 failure: v0.3.44 is deployed to all nine gateways, with the Linux/glibc arena cap enabled by default. The clean 1,800-second field unload reduced RSS from 2,903 to 565 MiB; a fresh workspace then loaded and
code_definitionsucceeded. Three additional consecutive default-allocator cycles each loaded a new workspace and evicted it with zero sessions, queries, commands, or workspaces afterward; post-unload RSS was 297, 1,216, and 294 MiB. The values vary and do not return to the 14 MiB cold-start baseline, but they did not grow monotonically and the next workspace loaded successfully after each cycle. All temporary allocator/thread-pool/idle-timer overrides were removed.The reported ~33 GiB retained RSS and refusal to admit another engine are resolved by the release. Closing #1590 with the measured warm-idle range recorded above.
alex09x commentedon Oct 29, 2026
Resolved and field-verified on v0.3.44. The issue documents the measured warm-idle RSS range and residual difference from cold start.