Repository navigation
fix(server): price a plan's NPU engines by driver residency - #975
Conversation
|
@wOvAN This PR reminds me that I may have overlooked a boundary/ownership issue in #964. The server now parses I'd prefer a generic resource-estimation interface: models/components interpret their execution plans and residency policies, the shared RKNN library reports individual engine requirements, and the server only applies its memory guard to the resulting estimate. I'd hold this PR while we address that boundary, including the existing logic from #964. |
|
Reworked in 2dc834a along the boundary you describe, including the
Board evidence for the new shape (RK3588, one cold server per row, six-rung ladder, weights 2153.1 MiB, Σ driver residency 10747 MiB): with the ladder ceiling applied the estimate is 7.09 GiB and with
|
4e00a9d to
241f830
Compare
|
@wOvAN Some issues:
|
|
Reworked all four points — force-push set 1. qwen3's custom loader reported nothing. Added a Before the hook the same call returned 2. The cap clipped the set below what the eviction rule admits. Applied budget is now 3. Plan engines found by the file scan were priced at the weights proxy. The guard now applies the reported plan first, and the weight/engine file scans skip every path the plan already counted (shared 4. The generic runtime was interpreting RKNN artifacts. The reader moved out of The duplicated budget knob is unified on #979's name: Validation after the rework: PC |
2d37310 to
1014af3
Compare
|
Rebased onto |
|
@wOvAN I made some changes to this PR directly. Could you test the RKNN part?
The current boundary is:
|
The load guard estimates a model's footprint from file sizes, so an `exec_plan` engine is priced at its image. On RK3588 every engine of the CosyVoice3 flow ladder is a 661-714 MB file that occupies 0.78-3.75 GiB once resident, so the estimate is wrong per rung in both directions: a 110-frame engine costs less than 1.5x its image, a 250-frame engine 2 GiB more than its image. Ask the driver instead, through the instrument the aperture ladder already uses: `engine_memory_mib`, which now has a non-RKNN stub returning 0 so the declaration is unconditional and a consumer needs no preprocessor test. Where the driver cannot say -- every build without an RKNN runtime, every file that is not an engine image -- the engine falls back to its file size, so a PC estimate is bit-identical to the old one and `server_config_test` passes unchanged. A ladder is priced by the sum of its declared engines, i.e. as if every rung were resident. The model evicts to stay inside the NPU aperture, whose ceiling is the model's policy and not knowable in the server, and undercharging is the failure this estimate has already been caught in once (0xShug0#964: 3.28 GiB estimated against 6.20 GiB declared).
The load guard read `exec_plan` JSON and split its artifact strings, and priced every declared engine as resident. Both are the model's knowledge: the plan's grammar belongs to the family that writes it, and so does the eviction rule that bounds the set (CosyVoice3's flow ladder shares one NPU aperture and evicts to stay inside it). `IVoiceModelLoader::model_resources()` now answers what a family will hold resident for one load -- the engines its plans declare, priced by the target runtime, plus the ceiling its own policy enforces on that set -- and `ModelRegistry::model_resources()` routes to the loader that owns the family. The plan value is read in one place, `runtime::exec_plan_engine_requirements()`, which resolves each artifact the way a sidecar resolves it and prices it with the shared RKNN library; a spec-backed family adds a residency policy by setting one field of its config. CosyVoice3 states its aperture ceiling through `rknn_aperture_budget_mib()`, the same reading that builds the ladder's `ApertureBudget`, so the number that evicts and the number that estimates cannot drift apart. The server keeps the parts that are its own: pricing at 1.0 rather than at the runtime factor (a compiled engine is resident at its own allocation, it builds no ggml graph), the file-size fallback where the runtime cannot answer, de-duplication against the weight scan, and the family ceiling applied to the priced set. It keeps no plan grammar: an `exec_plan` value is scanned as a path like any other option, where the JSON string is never a file, and the engines come from the loader.
`exec_plan_engine_requirements` is where the plan grammar now lives, and its only coverage so far is `server_config_test`, which is registered under `ENGINE_BUILD_EXTENDED_TESTS` and so never runs in CI. The reader's own contract -- a ladder packed into one artifact value, several components, an artifact that is not a string, an unset ladder, a value that is not JSON -- is the part that needs no board, so it goes into the RKNN seam gate that CI does run. Prices stay the driver's: in a build without an RKNN runtime every requirement reports 0, which is what the caller's fallback keys on.
The guard paragraph said the server reads the artifacts out of an `exec_plan`; it now asks the family's loader for the engines it will hold resident and applies the ceiling that family states.
The `exec_plan` cases asked the guard for engines a plan declares, and the guard asks the loader that owns the family -- so in a build that compiled neither CosyVoice3 nor Piper, every engine contributes nothing and the expected numbers were fiction. CI's sanitizer jobs are exactly that build: `-DAUDIOCPP_MODELS=builtin_audio_utils`, one loader, no `cosyvoice3`. The cases now price the engines when the family is in this registry and assert the empty plan when it is not, which is the same contract read from the other side: a family with no loader reports nothing.
…gine The generic runtime option surface was parsing RKNN execution plans: splitting comma-separated artifact lists and asking librknnrt what each engine costs. That is RKNN knowledge living in framework/runtime; move the reader to engine::rknn::exec_plan_engine_requirements in framework/rknn and have the spec-backed loader call it. Behavior is unchanged.
qwen3_tts has a custom loader, so it inherited the empty base
model_resources() and a declared RKNN speech decoder contributed nothing to a
consumer's residency estimate. Read qwen3_tts.exec_plan the same way the
spec-backed base does and report its engines priced by the driver.
Validated on RK3588 (ENGINE_ENABLE_RKNN=ON, runtime 2.3.2): registry
model_resources("qwen3_tts", ..., exec_plan of the two stateful vocoder
engines) reports engines=2 with resident_mib=263/331 from RKNN_QUERY_MEM_SIZE,
where the same call before this hook reported engines=0.
The residency budget accessor was added twice under two names across the stacked branches (rknn_aperture_budget_mib here, aperture_budget_mib with the AUDIOCPP_RKNN_APERTURE_MIB env knob there). Keep the latter, which the trace key already follows, so one knob exists after both land.
…gine Three changes to the load guard's estimate, from the review: - Apply the loader-reported resources before the weight-file scans and skip counted paths, so a declared engine found by both paths is priced once and at the driver's residency figure, not at the weights x1.5 proxy. - When a family reports a sharing budget, floor the applied cap at the largest single engine: the ladder's eviction rule admits one engine wider than the budget by evicting the rest, so min(aperture, cap) under-counted a set whose single widest rung exceeds the budget. - Extend the model_resources docblock with that floor contract, and document the reported-first, budget-floored flow in the server README. Tests cover the floor (sum fits under the default budget; a ceiling floors at the largest rung), the de-duplication (a plan engine and its file scan price once, at the engine figure when the family is compiled), and the qwen3 hook.
1743d57 to
8f64190
Compare
|
Tested your rewrite on the RK3588 board (aarch64, Unit tests on the board:
One real finding: a malformed plan now crashes the server at startup when the guard is on. The four Minimal fix along your boundary: |
|
@wOvAN Thanks for testing! I will impl the fix. |
|
PR updated. |
|
Tested The deferred-error path is what I re-ran the probe suite against, calling
And end to end on the server binary from the same rebuild: the config that killed the previous head at startup ( LGTM from my side. |
… Studio polish + Clang CI + prebuilt RK3588 engine packages Upstream commits fcb4619..503dfb3 (11): - LoRA adapter refactor + ComfyUI/LoKr for YuE2, VibeVoice, ACE-Step (0xShug0#974) - Studio layout polish, unified Studio/Arena voice selection (0xShug0#980) - C++17 structured-binding capture fix (0xShug0#986) - refactor(rknn): sliced graph through one shared helper (0xShug0#983) - Linux Clang C++17 CI build (0xShug0#987) - voxcpm2: cheaper AudioVAE decode (0xShug0#981) - refactor(tools/rknn): one report line/bootstrap per converter (0xShug0#990) - Higgs Audio voice preserved across reference-free text chunks (0xShug0#985) - fix(rknn): driver named in sidecar failure (0xShug0#989) - server: NPU engines priced by driver residency (0xShug0#975) - feat(model_spec): prebuilt RK3588 engines as rknn packages (0xShug0#991) # Conflicts: # CMakeLists.txt # src/models/higgs_audio_tts/session.cpp
Why
#964taught the load guard to read the engines anexec_plandeclares out of the plan's JSON, and priced them by file size on the stated ground thatlibrknnrtcannot report a weight-set size. It can:rknn_init(..., RKNN_FLAG_COLLECT_MODEL_INFO_ONLY, ...)plusrknn_query(RKNN_QUERY_MEM_SIZE)reportstotal_weight_size + total_internal_size, which is the allocation the real init goes on to make. That is the instrument#970put into the shared RKNN layer for the aperture ladder, and the guard is the second consumer it was written for.The boundary review on this PR moved the plan knowledge out of the server, which is where the shape below comes from.
What
IVoiceModelLoader::model_resources(model_path, session_options)(new, default{}) answers what a family will hold resident for one load: aModelResourcePlanofModelEngineRequirement{path, resident_mib}, priced by the target runtime, plusresident_cap_mib- the ceiling the family's own eviction rule enforces on that set (0 = no ceiling, the whole set is resident).ModelRegistry::model_resources(family, ...)routes to the loader that owns the family.runtime::exec_plan_engine_requirements()walks the components' artifact strings, comma-separated ladders included, resolves each name the way a sidecar resolves it (rknn::resolve_engine_path) and prices it with the shared RKNN library (rknn::engine_memory_mib, now declared outside#if defined(ENGINE_HAVE_RKNN)with a stub returning 0, so a consumer inapp/servercan ask in any build without its own preprocessor test). The spec-backed loader calls it, so a family adopts a residency policy by setting one field of its config (SpecBackedVoiceModelConfig::residency_cap_mib) rather than by teaching a consumer its plan layout.rknn_aperture_budget_mib()(declared inrknn_flow.h), the same reading that builds the ladder'sApertureBudget- the number that evicts and the number that estimates cannot drift apart.AUDIOCPP_RKNN_APERTURE_MIB(PR feat(rknn): let the board state its aperture ceiling #979) is read there, so that PR's reader is now subsumed by this one.exec_planvalue is scanned as a path like any other session option, where the JSON string is never a file, and the engines come from the loader.A ladder is priced by the sum of its engines, capped by what the family says it keeps. Sum-then-cap is deliberate: the ceiling is measured per board and per engine set (
#970), and undercharging is the failure this estimator has already been caught in.Validation
PC (x86_64,
cmake -S . -B build-pc -DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_EXTENDED_TESTS=ON -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=piper_tts,kokoro_tts,qwen3_tts,voxcpm1,voxcpm2,moss_tts_v15,f5_tts,echo_tts,fish_audio,glm_tts,dots_tts,mira_tts,sanotts,cosyvoice3):server_config_testandrknn_engine_testpass;python3 tools/check_loader_catalog_sync.pypasses.server_config_testnow pins the new contract: engines of a plan resolved relative to the model directory and priced at 1.0 (not the runtime factor), a family with no ceiling priced by the sum of a ladder, a family that states a ceiling priced by that ceiling (the CosyVoice3 case, ceiling set through its knob: three 1 GiB rungs -> 1024 MiB), an engine that the checkpoint-tree scan already counted not counted twice, an absent engine contributing nothing, and a malformed plan contributing nothing.Board (Orange Pi 5 Plus, RK3588, 31 GiB, native aarch64 build with
-DENGINE_ENABLE_RKNN=ON -DENGINE_RKNN_INCLUDE_DIR=/usr/local/include -DENGINE_RKNN_LIBRARY=/usr/lib/librknnrt.so), one coldaudiocpp_serverper row,--backend cpu --threads 4 --min-free-memory-mb <n>, onezero_shotrequest (seed: 1234) againstcosyvoice3-q8_0.gguf(2153.1 MiB) with the six-rungexec_planladderflow_estimator_seq{110,200,250,400,500,900}.rknn:AUDIOCPP_RKNN_APERTURE_MIB--min-free-memory-mbrc=503rc=200, WAVa4346f68a7760(accounting off)rc=5030(accounting off)rc=503Verbatim:
cannot load model 'cosyvoice3': estimated 7.09 GiB + 40000 MiB headroom exceeds available host memory (29.76 GiB)andestimated 13.77 GiB + 20000 MiB headroom exceeds available host memory (29.76 GiB). One binary (68b2b589), one config; the ceiling is the only difference between rows, and it flips the guard's decision at the same headroom. The six engines' driver-reported residency sums to 10747 MiB (786/1129/1426/2615/3752/1039) against 4076 MiB of images, so the ceiling is what keeps a 2.1 GiB model loadable on a 31 GiB board.Audio is unchanged by the patch: the pre-change binary (
d982036a, engines priced at file x 1.5) on the same config, same request and--min-free-memory-mb 18000rendersa4346f68a776, bit-identical to the new binary's row above.Limits
The guard spends about 0.5 s per declared engine at load (
rknn_initin info-only mode) - a few seconds once per startup on a six-rung ladder, which is the cost the ladder already pays per cold rung, moved earlier. The guard still only samples host memory; it has never tracked NPU apertures, and it trusts the ceiling the family states, so a family that states none is priced by its sum.