Skip to content

npu plan: a plan that names an op with no kernel here is refused at load, naming the design (rule (c), engine side) - #1420

Open
bong-water-water-bong wants to merge 1 commit into
mainfrom
1bit/npu-plan-completeness
Open

bong-water-water-bong wants to merge 1 commit into
mainfrom
1bit/npu-plan-completeness

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What — rule (c), engine side

Owner: "plan.json load refuses up front, naming the reason, any op without a kernel for its shape/dtype … never a silent host fallback."

A plan can be complete as JSON and unservable in fact: designs[].elf is a name, and whether that file exists is a different question from whether the plan is well formed. Until now the answer arrived later — the target's own probe inside open_prefill, after the setup work. Plan::check_artifacts() asks it at load instead:

plan designs[1] (O): the artifact "full_i8_O_K2048_N1024.elf" is in none of 1 kernel dir(s) — this op has no kernel here

naming the design index, the role and the artifact. It is wired into init_impl before any context is opened, and the target's probe stays as the second line rather than being replaced: this one runs earlier and says more.

Deliberately not checked here: whether a weight tensor carries its quantisation. That is the container's question (npu/q4nx.h refuses a tensor it cannot decode), and this layer only sees the plan — inventing a second opinion about tensor quantisation is how two answers start disagreeing. Stated in the header where the function is declared.

Tests (no device, temp dirs)

case expectation
both artifacts present complete
one missing refused, and the message names designs[1], O and the elf
a second directory searched first-hit, like the forward's own order
a zero-byte file not a kernel (the same >= 1 byte rule the forward's probe uses)
a plan with no designs section complete — nothing named, nothing missing
an empty directory list refused with "no kernel directory" rather than crashing

Rows

row verdict
npu_plan (the new cases, plus every earlier one) PASS — npu_plan: ok
ctest -R "npu_plan|designs_parity|design_grid|op_reference" 6/6
build Release, host-only tree (build-plan) — this change is device-free by construction; the NPU build in my scratch worktree is currently blocked by a stale ONEBIT_NPU_RT_LIB path, which is an environment issue I am reporting rather than working around
tools/copyright.py --check rc 0

Refs #799.

…oad, naming the design (rule (c), engine side)

Owner's rule (c): 'plan.json load refuses up front, naming the reason, any op without a kernel for its
shape/dtype ... never a silent host fallback.'

The plan can be complete as JSON and unservable in fact: designs[].elf is a name, and whether that file is
in the kernel directory is a different question from whether the plan is well formed. Until now the answer
arrived later -- the target's own probe, at open_prefill, after the setup work. Plan::check_artifacts()
asks it at load instead: for each named design, the artifact must be present and non-empty in one of the
given kernel directories (the forward's own >= 1 byte rule), and a miss refuses with

  plan designs[1] (O): the artifact "full_i8_O_K2048_N1024.elf" is in none of 1 kernel dir(s) -- this op
  has no kernel here

which names the design, the role and the artifact. Wired into init_impl before any context is opened, and
the target's probe stays as the second line rather than being replaced: this one runs earlier and says more.

Deliberately not checked here: whether a weight tensor carries its quantization. That is the container's
question (npu/q4nx.h refuses a tensor it cannot decode); this layer only sees the plan, and inventing a
second opinion about tensor quantization is how two answers start disagreeing.

Tests (tests/npu_plan_test.cpp, no device, temp dirs): both artifacts present is complete; a missing one is
refused and the message names designs[1]/O/elf; a second directory is searched first-hit like the forward;
a zero-byte file is not a kernel; a plan with no designs section is complete (nothing named, nothing
missing); and an empty directory list refuses with 'no kernel directory' rather than crashing.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#1420 ready for review at 6a437e1b — rule (c) engine side, and it is small on purpose.

Plan::check_artifacts(dirs, err): for each designs[] entry the artifact must be present and non-empty in one of the given kernel directories (the forward's own >= 1 byte rule, first hit wins), and a miss refuses at load with plan designs[1] (O): the artifact "…" is in none of 1 kernel dir(s) — this op has no kernel here. Wired into init_impl before any context is opened; the target's probe stays as the second line rather than being replaced.

Two boundary calls worth your eye:

  • Tensor quantisation is not checked here, and the header says why: that is the container's question (npu/q4nx.h refuses a tensor it cannot decode) and this layer only sees the plan. A second opinion about tensor quantisation is how two answers start disagreeing.
  • A plan with no designs section is complete — nothing named, nothing missing — which is what every plan written before the field means. With rule (b) slice 1 in review (npu plan: the registration rule and its test for op references (rules (b), slice 1) #1409) the two rules meet when the loader clause lands.

Rows: npu_plan: ok with six new cases (both present / one missing / second dir / zero-byte file / no section / empty dir list), ctest -R "npu_plan|designs_parity|design_grid|op_reference" 6/6, copyright.py --check rc 0.

One environment note rather than a claim: this is a host-only change and I ran it in a host-only build, because my scratch worktree's NPU configure now fails on a stale ONEBIT_NPU_RT_LIB path (/home/bcloud/wt/bonsai-speed-build/libonebit_npu_rt.a no longer exists; the current drops are under ~/lb/*/libonebit_npu_rt.a). Nothing about this PR needs the NPU build, but the lane should not pretend it ran one.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6a437e1b6839c2cb862cc33450daf0eb0ef05f45

Plan::check_artifacts() refuses a plan whose designs[] names an ELF that isn't on disk. It runs at load, before any context is opened, and names the design index, role and artifact. That's rule (c) for the part the plan layer can see. The scope it leaves to the container (tensor quantisation, npu/q4nx.h) is the right split, and it's stated where the function is declared.

The wiring checks out:

  • it is skipped under host_gemm_ (offline mode opens no ELF, so a missing artifact isn't an error there, consistent with open_prefill);
  • it only runs when a plan is loaded;
  • it searches {dir}, the same directory open_prefill takes from setup_.kernels_dir, so the two can't disagree on where to look;
  • the >= 1 byte rule matches the forward's probe;
  • an empty designs section passes, so every plan written before the section still loads.

What I ran at this head (base 336fc786, on main; NPU build, thermal-run.sh):

  • npu_plan_test, npu_plan_op_reference_test and onebit_npu_forward: built clean;
  • ctest -R "npu_plan|designs_parity|design_grid|op_reference": 6/6. npu_plan_op_reference shows Not Run until its own target is built;
  • tools/copyright.py --check: rc 0.

One latent nit, not reachable today: open_prefill deliberately skips the two pb16i split roles (role_is_pb16i_split; QKVFAQ/QKVFAKV, added for GDN packs at q4nx_forward.cpp:1037), because they "exist only as pb16i kernels … and no int8 ELF on disk". check_artifacts doesn't know that. A plan whose designs lists those two roles under their derived int8 ELF names would be refused for artifacts the forward never opens. Today nothing writes such a plan: tools/npu_plan.py's designs_from_config returns None for GDN configs (linear_num_key_heads). But designs_agree requires the section to match the derived set including those roles, so the first GDN plan with a designs section would hit this. Skipping the QKVFA roles when the pack is pb16i, or checking the ELF the target will actually open, would close it ahead of time.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant