Repository navigation
fix(server): count exec_plan engines in the free-memory guard - #964
Merged
Merged
Conversation
The load guard estimates what a model will occupy and refuses a load that does not fit (`--min-free-memory-mb`, HTTP 503 `insufficient_memory`). It scans `session_options` values as paths, which is right for the auxiliary files it was written for and useless for `exec_plan`: the value is a JSON object, never a file, so every engine a plan declared contributed 0 bytes. On RK3588 that is not a rounding error -- a `cosyvoice3` model with the three-engine flow ladder estimates 3.28 GiB while the files it will map are 6.20 GiB, so the guard waves through a load that is 2.92 GiB bigger than it approved. Parse the plan instead of stat-ing it: for an option whose key ends in `.exec_plan`, walk each component's `artifacts` strings, split on commas (CosyVoice3's ladder packs the ladder into one value), resolve each relative path against the same base the auxiliary scan uses -- the model directory, which is also the root `engine::rknn::resolve_engine_path` joins onto -- and count the file. A malformed plan contributes nothing, because the plan is the session's error to report, not the guard's. Counting goes through a set of canonical paths, so an engine that sits inside a checkpoint tree the estimator already sums is counted once. Validation: - `server_config_test` (built with `-DENGINE_BUILD_EXTENDED_TESTS=ON`, the gate its target sits behind): four new estimator cases -- engines named relative to the model directory plus one absolute path, an engine inside a summed tree counted once with an absent engine adding nothing, and a malformed plan contributing nothing. Passes; with the estimator change reverted the suite fails on `exec_plan engines resolve relative to the model directory, and an absolute path is used as given`. The absolute-path case is spelled with `generic_string()` because the first CI pass failed on `windows-2022` alone: a native Windows temp path spliced into the plan string carries backslashes, which are invalid JSON escapes -- the parser says `failed to parse json at byte 8` -- so the plan decoded to nothing and the declared engines counted 0. - Host: `cmake --build build-pc --target server_config_test audiocpp_server` clean; `tools/check_loader_catalog_sync.py --self-test && tools/check_loader_catalog_sync.py` -> ok. - Board (RK3588, Orange Pi 5 Plus, native build, `cosyvoice3` on a GGUF with three engines declared by bare name, guard on): refuses at `--min-free-memory-mb 25000` with `estimated 6.20 GiB + 25000 MiB headroom exceeds available host memory (29.75 GiB)`; the same server built without the estimate change loads that request (HTTP 200) and refuses only at 27500, reporting `estimated 3.28 GiB`. The 2.92 GiB delta is the ladder: 2094213284 bytes of declared engines at the estimator's 1.5x runtime factor.
wOvAN
force-pushed
the
rknn-guard-exec-plan-engines
branch
from
October 18, 2026 00:51
039efb1 to
1e09e26
Compare
Owner
|
@wOvAN Thanks! PR merged. |
wOvAN
pushed a commit
to wOvAN/audio.cpp
that referenced
this pull request
Oct 18, 2026
The load guard estimates a model's footprint from file sizes, so an `exec_plan` engine is priced at its image. On RK3588 every engine of the CosyVoice3 flow ladder is a 661-714 MB file that occupies 0.78-3.75 GiB once resident, so the estimate is wrong per rung in both directions: a 110-frame engine costs less than 1.5x its image, a 250-frame engine 2 GiB more than its image. Ask the driver instead, through the instrument the aperture ladder already uses: `engine_memory_mib`, which now has a non-RKNN stub returning 0 so the declaration is unconditional and a consumer needs no preprocessor test. Where the driver cannot say -- every build without an RKNN runtime, every file that is not an engine image -- the engine falls back to its file size, so a PC estimate is bit-identical to the old one and `server_config_test` passes unchanged. A ladder is priced by the sum of its declared engines, i.e. as if every rung were resident. The model evicts to stay inside the NPU aperture, whose ceiling is the model's policy and not knowable in the server, and undercharging is the failure this estimate has already been caught in once (0xShug0#964: 3.28 GiB estimated against 6.20 GiB declared).
wOvAN
pushed a commit
to wOvAN/audio.cpp
that referenced
this pull request
Oct 19, 2026
The load guard estimates a model's footprint from file sizes, so an `exec_plan` engine is priced at its image. On RK3588 every engine of the CosyVoice3 flow ladder is a 661-714 MB file that occupies 0.78-3.75 GiB once resident, so the estimate is wrong per rung in both directions: a 110-frame engine costs less than 1.5x its image, a 250-frame engine 2 GiB more than its image. Ask the driver instead, through the instrument the aperture ladder already uses: `engine_memory_mib`, which now has a non-RKNN stub returning 0 so the declaration is unconditional and a consumer needs no preprocessor test. Where the driver cannot say -- every build without an RKNN runtime, every file that is not an engine image -- the engine falls back to its file size, so a PC estimate is bit-identical to the old one and `server_config_test` passes unchanged. A ladder is priced by the sum of its declared engines, i.e. as if every rung were resident. The model evicts to stay inside the NPU aperture, whose ceiling is the model's policy and not knowable in the server, and undercharging is the failure this estimate has already been caught in once (0xShug0#964: 3.28 GiB estimated against 6.20 GiB declared).
wOvAN
pushed a commit
to wOvAN/audio.cpp
that referenced
this pull request
Oct 19, 2026
The load guard estimates a model's footprint from file sizes, so an `exec_plan` engine is priced at its image. On RK3588 every engine of the CosyVoice3 flow ladder is a 661-714 MB file that occupies 0.78-3.75 GiB once resident, so the estimate is wrong per rung in both directions: a 110-frame engine costs less than 1.5x its image, a 250-frame engine 2 GiB more than its image. Ask the driver instead, through the instrument the aperture ladder already uses: `engine_memory_mib`, which now has a non-RKNN stub returning 0 so the declaration is unconditional and a consumer needs no preprocessor test. Where the driver cannot say -- every build without an RKNN runtime, every file that is not an engine image -- the engine falls back to its file size, so a PC estimate is bit-identical to the old one and `server_config_test` passes unchanged. A ladder is priced by the sum of its declared engines, i.e. as if every rung were resident. The model evicts to stay inside the NPU aperture, whose ceiling is the model's policy and not knowable in the server, and undercharging is the failure this estimate has already been caught in once (0xShug0#964: 3.28 GiB estimated against 6.20 GiB declared).
0xShug0
added a commit
that referenced
this pull request
Oct 19, 2026
* fix(server): price a plan's NPU engines by driver residency The load guard estimates a model's footprint from file sizes, so an `exec_plan` engine is priced at its image. On RK3588 every engine of the CosyVoice3 flow ladder is a 661-714 MB file that occupies 0.78-3.75 GiB once resident, so the estimate is wrong per rung in both directions: a 110-frame engine costs less than 1.5x its image, a 250-frame engine 2 GiB more than its image. Ask the driver instead, through the instrument the aperture ladder already uses: `engine_memory_mib`, which now has a non-RKNN stub returning 0 so the declaration is unconditional and a consumer needs no preprocessor test. Where the driver cannot say -- every build without an RKNN runtime, every file that is not an engine image -- the engine falls back to its file size, so a PC estimate is bit-identical to the old one and `server_config_test` passes unchanged. A ladder is priced by the sum of its declared engines, i.e. as if every rung were resident. The model evicts to stay inside the NPU aperture, whose ceiling is the model's policy and not knowable in the server, and undercharging is the failure this estimate has already been caught in once (#964: 3.28 GiB estimated against 6.20 GiB declared). * refactor(server): let a model report the engines its plan needs The load guard read `exec_plan` JSON and split its artifact strings, and priced every declared engine as resident. Both are the model's knowledge: the plan's grammar belongs to the family that writes it, and so does the eviction rule that bounds the set (CosyVoice3's flow ladder shares one NPU aperture and evicts to stay inside it). `IVoiceModelLoader::model_resources()` now answers what a family will hold resident for one load -- the engines its plans declare, priced by the target runtime, plus the ceiling its own policy enforces on that set -- and `ModelRegistry::model_resources()` routes to the loader that owns the family. The plan value is read in one place, `runtime::exec_plan_engine_requirements()`, which resolves each artifact the way a sidecar resolves it and prices it with the shared RKNN library; a spec-backed family adds a residency policy by setting one field of its config. CosyVoice3 states its aperture ceiling through `rknn_aperture_budget_mib()`, the same reading that builds the ladder's `ApertureBudget`, so the number that evicts and the number that estimates cannot drift apart. The server keeps the parts that are its own: pricing at 1.0 rather than at the runtime factor (a compiled engine is resident at its own allocation, it builds no ggml graph), the file-size fallback where the runtime cannot answer, de-duplication against the weight scan, and the family ceiling applied to the priced set. It keeps no plan grammar: an `exec_plan` value is scanned as a path like any other option, where the JSON string is never a file, and the engines come from the loader. * test(rknn): cover the plan reader outside an extended gate `exec_plan_engine_requirements` is where the plan grammar now lives, and its only coverage so far is `server_config_test`, which is registered under `ENGINE_BUILD_EXTENDED_TESTS` and so never runs in CI. The reader's own contract -- a ladder packed into one artifact value, several components, an artifact that is not a string, an unset ladder, a value that is not JSON -- is the part that needs no board, so it goes into the RKNN seam gate that CI does run. Prices stay the driver's: in a build without an RKNN runtime every requirement reports 0, which is what the caller's fallback keys on. * docs(server): describe the estimate the model reports The guard paragraph said the server reads the artifacts out of an `exec_plan`; it now asks the family's loader for the engines it will hold resident and applies the ceiling that family states. * test(server): ask the registry which families a build compiled The `exec_plan` cases asked the guard for engines a plan declares, and the guard asks the loader that owns the family -- so in a build that compiled neither CosyVoice3 nor Piper, every engine contributes nothing and the expected numbers were fiction. CI's sanitizer jobs are exactly that build: `-DAUDIOCPP_MODELS=builtin_audio_utils`, one loader, no `cosyvoice3`. The cases now price the engines when the family is in this registry and assert the empty plan when it is not, which is the same contract read from the other side: a family with no loader reports nothing. * refactor(rknn): keep the exec-plan artifact reader beside the RKNN engine The generic runtime option surface was parsing RKNN execution plans: splitting comma-separated artifact lists and asking librknnrt what each engine costs. That is RKNN knowledge living in framework/runtime; move the reader to engine::rknn::exec_plan_engine_requirements in framework/rknn and have the spec-backed loader call it. Behavior is unchanged. * fix(qwen3): report the exec plan's engines through the resources hook qwen3_tts has a custom loader, so it inherited the empty base model_resources() and a declared RKNN speech decoder contributed nothing to a consumer's residency estimate. Read qwen3_tts.exec_plan the same way the spec-backed base does and report its engines priced by the driver. Validated on RK3588 (ENGINE_ENABLE_RKNN=ON, runtime 2.3.2): registry model_resources("qwen3_tts", ..., exec_plan of the two stateful vocoder engines) reports engines=2 with resident_mib=263/331 from RKNN_QUERY_MEM_SIZE, where the same call before this hook reported engines=0. * refactor(cosyvoice3): spell the aperture budget knob by one name The residency budget accessor was added twice under two names across the stacked branches (rknn_aperture_budget_mib here, aperture_budget_mib with the AUDIOCPP_RKNN_APERTURE_MIB env knob there). Keep the latter, which the trace key already follows, so one knob exists after both land. * fix(server): price reported engines first and floor the cap at one engine Three changes to the load guard's estimate, from the review: - Apply the loader-reported resources before the weight-file scans and skip counted paths, so a declared engine found by both paths is priced once and at the driver's residency figure, not at the weights x1.5 proxy. - When a family reports a sharing budget, floor the applied cap at the largest single engine: the ladder's eviction rule admits one engine wider than the budget by evicting the rest, so min(aperture, cap) under-counted a set whose single widest rung exceeds the budget. - Extend the model_resources docblock with that floor contract, and document the reported-first, budget-floored flow in the server README. Tests cover the floor (sum fits under the default budget; a ceiling floors at the largest rung), the de-duplication (a plan engine and its file scan price once, at the engine figure when the family is compiled), and the qwen3 hook. * Keep resource selection and residency policy model-owned * Defer invalid plan errors to sessions and fix canonical path tests --------- Co-authored-by: wOvAN <wovan.dev@protonmail.com> Co-authored-by: 0xShug0 <231717474+0xShug0@users.noreply.github.com>
cunba-ai
pushed a commit
to cunba-ai/audio.cpp
that referenced
this pull request
Oct 28, 2026
…ke-word/Whistle NEON + Drama3 deepening + per-model server device selection Upstream commits a8ba5a3..5074956 (16): - shared object libraries declared in model registrations (0xShug0#966) - OpenWakeWord Alexa GGUF package (0xShug0#968) - Whistle host decoder restored + opt-in NEON acceleration (0xShug0#967) - opt-in NEON network execution for microWakeWord (0xShug0#969) - fish_drama3: per-layer expert counts, speech-only head (0xShug0#959) - server: exec_plan engines counted in free-memory guard (0xShug0#964) - server: config model settings kept on WebUI loads, demo voices with --ui (0xShug0#971) - fix(rknn): exec_plan engine lookup next to the model (0xShug0#963) - webui: configured models treated same with/without UI management (0xShug0#972) - perf(rknn): ladder engine priced by driver report (0xShug0#970) - feat(server): per-model backend device selection (0xShug0#973) - perf(fish_dac): chunked reference encoding cuts VRAM peak (0xShug0#960) - webui: Vietnamese interface translations (0xShug0#839) - Drama3 base/small release cleanup (0xShug0#976) - text chunking: closing quotes kept with their sentence, tag-aware (0xShug0#977) - fish_drama3: default top_k 4097 no cap, matching Fish's Space (0xShug0#978) # Conflicts: # src/community_models/whistle_asr/session.cpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The free-memory load guard (
--min-free-memory-mb, HTTP 503insufficient_memory) estimates what a model will occupy and refuses a load that does not fit. It scans everysession_optionsvalue as a path — right for the auxiliary files it was written for, useless forexec_plan: that value is a JSON object, never a file, so every compiled engine a plan declares contributes 0 bytes to the estimate.On RK3588 this is not a rounding error.
cosyvoice3on a GGUF with the three-engine flow ladder declared:The guard approves a load that is 2.92 GiB bigger than what it just approved.
How
Parse the plan instead of
stat-ing it. For an option whose key ends in.exec_plan: walk each component'sartifactsstrings, split on commas (CosyVoice3 packs its ladder into one value), resolve each relative path against the same base the auxiliary scan already uses — the model directory, which is also the rootengine::rknn::resolve_engine_pathjoins onto — and count the file.cosyvoice3,qwen3_tts,kokoro_tts,piper_tts,inflect_v2) are covered by the one site.Validation
Host (x86_64):
server_config_test(the suite is registered behindENGINE_BUILD_EXTENDED_TESTS) gains four estimator cases: engines named relative to the model directory plus one absolute path; an engine inside a summed tree counted once, with an absent engine adding nothing; a malformed plan contributing nothing. Negative control — withapp/server/model_memory.cppreverted and the new cases kept, the suite fails onexec_plan engines resolve relative to the model directory, and an absolute path is used as given.tools/check_loader_catalog_sync.py --self-test && tools/check_loader_catalog_sync.py→ ok.Board — Orange Pi 5 Plus (RK3588), native build,
cosyvoice3onmodels/CosyVoice3-GGUF/cosyvoice3-q8_0.ggufwith"main": "flow_estimator_seq110.rknn,flow_estimator_seq200.rknn,flow_estimator_seq250.rknn"(bare names, symlinks in the GGUF directory), one cold server per run, guard on, one request:--min-free-memory-mbcannot load model 'cosyvoice3': estimated 6.20 GiB + 25000 MiB headroom exceeds available host memory (29.75 GiB)RIFF…WAVEfmt, 67 630+ bytes) — the load the guard waved throughestimated 3.28 GiB + 27500 MiB headroom exceeds available host memory (29.75 GiB)The 2.92 GiB delta is exactly the ladder: 693499212 + 700834252 + 699879820 = 2094213284 bytes of declared engines at the estimator's 1.5× runtime factor.
Docs:
app/server/README.md, the paragraph describing what the load estimate covers.Limits
rknn_query(RKNN_QUERY_MEM_SIZE)reportstotal_weight_size/total_internal_sizeper engine, and a dummy init gets it without running the network — but it is a runtime call from inside a loaded session, while the guard is a pre-load estimate over paths on disk. Counting declared engine files is what the guard can do before a session exists; reading the engine's own size report belongs to the resident-footprint accounting the RKNN sidecar does (the aperture budget incosyvoice3/rknn_flow.cpp), and is where that number should go.