fix(checkpoint): validate FTW v1 indexes eagerly

[Avicennasis]

Problem. A malformed or truncated FTW v1 index (missing field, wrong type, out-of-range offset, duplicate key, shard shorter than declared) surfaces late, as an mmap or O_DIRECT failure inside the load, with no indication of which index field was wrong.

Change. Parse and validate the index once at FTWReader open: required fields, JSON integers within the signed 64-bit range, duplicate keys rejected, entry names checked, tensors and shards checked for coverage and overlap, each shard stat'd and size-checked. Errors raise FTWFormatError (a ValueError) naming the field. The writer validates its own metadata.

Shard symlinks are followed — Hugging Face hub snapshots are symlink farms — and the regular-file and size checks apply to the link target. A shard name that is not a filesystem name (a JSON-escaped lone surrogate) is refused under the same error type rather than escaping as UnicodeEncodeError.

Nothing the writer produces is rejected; existing valid checkpoints load unchanged.

Not in this PR. Runtime integrity re-checks, admission, generations, and bank-schema validation are separate changes.

Tested. Ubuntu 24.04 x86_64, kernel 7.0.0, 16 cores / 125 GiB, no GPU — this is a pure-Python change and the added tests are CPU-only. Python 3.12, torch 2.11.0+cu130, transformers 5.17.0, pytest 9.1.1, safetensors 0.8.0. Run with PYTHONPATH=python (an editable install needs CUDA_HOME); the C++ extension is not built in this environment, which accounts for one of the pre-existing failures below.

Focused:

PYTHONPATH=python python -m pytest -q tests/checkpoint/test_ftw_validation.py
85 passed

Also under python -O (85 passed), since several of the checks would be lost if they were assertions.

Full suite from the repo root, this branch:

PYTHONPATH=python python -m pytest -q tests
4 failed, 1346 passed, 427 skipped

and main at 9535656 in the same venv, same command:

4 failed, 1261 passed, 427 skipped

Identical failure sets — tests/engine/test_cache_budget.py::test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend, tests/engine/test_cache_budget.py::test_adjust_config_resolves_num_tokens_generic, tests/moe/test_offload.py::test_adjust_config_converts_moe_cache_rate_to_cache_size, and tests/kernels/test_swiglu_clamp.py::test_cpu_extension_supports_swiglu_clamp (the CPU extension is not built here). The +85 is this PR's test file.

No checkpoint was converted; the tests build their own FTW files with FTWWriter.

git diff --check clean; compileall ok; ruff 0.15.0 reports nothing on the four touched files (and nothing on their main counterparts).

Fixes #458.

[Avicennasis]

Rebased onto current main (cac247a, 0.1.3). #486 touched checkpoint/ftw.py underneath this — it added ftw_tensor_names(), a keep(name) predicate on iter_ftw_weights, and the Callable import — and all of that merged cleanly around the eager _validate_ftw_v1_index; the one conflict was the __all__ list, where #486 exports ftw_tensor_names and this PR exports FTWFormatError, so both are listed now. A git range-diff against the previous head shows nothing else moving.

#486's tests/checkpoint/test_ftw_hotfix_vision.py and test_ftw_weights.py now sit next to test_ftw_validation.py. iter_ftw_weights still opens the FTWReader (and so validates the index) before keep is consulted, so a malformed index is refused before any name is filtered.

Re-ran on a CPU-only Ubuntu 24.04 x86_64 box (AMD Ryzen 7 5800X, 16 threads, 125 GiB, no GPU; Python 3.12, torch 2.11.0+cu130, transformers 5.17.0, pytest 9.1.1, run with PYTHONPATH=python since an editable install needs CUDA_HOME):

• tests/checkpoint (all three files): 102 passed, 6 skipped, normally and under python -O; main at the same commit gives 17 passed, 6 skipped, so the +85 is this PR's tests. The 6 skips are fix(checkpoint): serve and repair FTW checkpoints around the vision encoder #486's FREETOKEN_*_MODEL not set cases.
• Full pytest -q tests from the repo root: this branch 8 failed / 1447 passed / 488 skipped, against main at cac247a in the same venv: 8 failed / 1362 passed / 488 skipped. Identical failure sets — tests/engine/test_cache_budget.py ×2, tests/moe/test_offload.py ×1, tests/kernels/test_swiglu_clamp.py::test_cpu_extension_supports_swiglu_clamp (the C++ extension is not built in this venv), three tests/mm/test_processor.py cases (No module named 'PIL'; Pillow is not in this venv), and tests/models/test_quant_config.py::…[bert-base-uncased] (the only snapshot in this box's HF cache; it yields zero probe modules). A first full run of the branch also tripped tests/daemon/test_daemon_serve_manager.py::test_crash_is_reaped_and_reported_without_stop — the log-ring line landing after the status flip, on a box under heavy load; it passes 10/10 in isolation on both trees and did not recur on a second full run.

Nothing in the change touches a kernel or a model family; the validator runs once at FTWReader open, on the index alone, which is why a CPU-only box is an honest environment for it.

[Avicennasis]

Rebased onto current main (2d8b9cc). git merge-tree named one conflicting file, checkpoint/ftw.py: #601 (NVFP4 / MXFP4 experts served fused through the expert banks) moved the row-layout split out of load_ftw_banks into a new _split_bank_entries helper that load_ftw_banks and the new load_ftw_banks_to_device both call, and this PR had changed the two asserts in that block into FTWFormatError. The resolution keeps #601's shape: the block stays in _split_bank_entries, where the two checks now raise FTWFormatError instead of asserting (the callers already close the reader on any exception, so the explicit reader.close() this PR used to carry there is not needed). __all__ lists both FTWFormatError and #601's load_ftw_banks_to_device. git range-diff against the previous head shows nothing else moving; the diffstat is the same four files, +1052/-24 (two lines fewer than before, the moved checks).

Re-ran on a CPU-only Ubuntu 24.04 x86_64 box (AMD Ryzen 7 5800X, 16 threads, 125 GiB, no GPU; Python 3.12.3, torch 2.11.0+cu130, transformers 5.16.1, safetensors 0.8.0, flashlib 0.3.0, pytest 9.1.1; run with PYTHONPATH=python, no C++ extension, no flashinfer):

• tests/checkpoint (all three files): 107 passed, 7 skipped, normally and under python -O; main at 2d8b9cc gives 22 passed, 7 skipped, so the +85 is this PR's test_ftw_validation.py. The 7 skips are the FREETOKEN_*_MODEL not set cases.
• Full pytest -q tests from the repo root: this branch 8 failed / 1601 passed / 635 skipped (2153 collected, 7:58), against main at 2d8b9cc in the same venv 8 failed / 1516 passed / 635 skipped (9:20). Identical failure sets, all environment-bound: tests/engine/test_cache_budget.py ×2 and tests/moe/test_offload.py ×1 (the fi attention backend needs flashinfer), tests/kernels/test_swiglu_clamp.py::test_cpu_extension_supports_swiglu_clamp and tests/models/deepseek_v41/test_engram.py::test_ftw_side_files_carry_the_tables_and_reopen (the C++ / row-store extensions are not built here), and three tests/models/test_quant_config.py cases that probe the snapshots in this box's HF cache. A first full run of the branch also tripped tests/daemon/test_daemon_serve_manager.py::test_crash_is_reaped_and_reported_without_stop (the exited with code 1 log line landing after the status flip; the box was at load average 90 with other suites running) -- the same case as on the previous rebase; it passes 10/10 in isolation on both trees and did not recur on the second full run, which is the one quoted.

Nothing in the change touches a kernel or a model family; the validator runs once at FTWReader open, on the index alone.