Repository navigation
Conversation
_pin_budget_bytes() returned 40% of physical RAM (25.1 GiB on the test box, ~25x above the real cumulative CUDA pin ceiling), so split residency never engaged and loads died in cudaHostRegister. Now on WSL2 it measures the ceiling once (lru_cache): cudaHostAlloc in 256 MiB chunks until the driver refuses, faults the pages to match pin-after-fill banks, returns 80% of the measured total for headroom, and frees the probe buffers. Plain Linux (no "microsoft" kernel tag) stays uncapped (None); FREETOKEN_PIN_BUDGET_GB still overrides. The refused alloc leaves a *sticky* CUDA error every later call replays as "out of memory"; cudaGetLastError() clears it before return. Experiment env: WSL2 kernel 6.18.33.2-microsoft-standard-WSL2, CUDA 13.0 (torch, libcudart.so.13), RTX 3090 24 GiB, 62.8 GiB RAM. Probed wall 0.75-1.0 GiB cumulative (256 MiB chunks); host_ptr_identity (UVA) False.
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 3, 2026
On some WSL hosts the driver refuses pinning after a few GiB, far below the 40%-of-RAM estimate, so the split residency never engaged and the load died in cudaHostRegister. Pin 256 MiB blocks once until the driver refuses or the estimate is reached, free them, and budget 80% of it. Alternative to upstream PR FlashML-org#233: probes up to the old estimate instead of a fixed 8 GiB (which would cap large hosts at 6.4 GiB), returns 0 rather than "uncapped" when nothing pins, allocates through the pinned extension that now clears the driver's latched error (FlashML-org#144), and keeps main's reserved-bytes signature. Co-authored-by: kitty <2165990891@qq.com> Assisted-by: Claude Opus 5.5
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 3, 2026
Logs the low-RAM loading batch pulled forward: FlashML-org#334, FlashML-org#233, FlashML-org#144 and FlashML-org#56. Assisted-by: Claude Opus 5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_pin_budget_bytes()返回物理内存的 40%(测试机 25.1 GiB),比真实累计 CUDA pin 上限高出约 25 倍,导致 split-residency 机制从不触发,模型加载时直接死在cudaHostRegister。现在在 WSL2 上,启动时一次性实测上限(lru_cache):用
cudaHostAlloc以 256 MiB 块逐步分配直到驱动拒绝,触页后与 pin-after-fill bank 行为一致,取实测值的 80% 留余量,并释放探针缓冲。真 Linux(内核无microsoft标签)保持不设限(返回None);FREETOKEN_PIN_BUDGET_GB环境变量仍可覆盖。被拒绝的分配会在 CUDA context 留下粘性错误,后续每次调用都会重放为 "out of memory";返回前用
cudaGetLastError()清除。实验环境:WSL2 kernel
6.18.33.2-microsoft-standard-WSL2,CUDA 13.0(torch, libcudart.so.13),RTX 3090 24 GiB,62.8 GiB RAM。实测 pin 墙 0.75–1.0 GiB 累计(256 MiB 块);host_ptr_identity(UVA)为 False。🤖 Generated with Claude Code