Repository navigation
fix(executorch): ExecuTorch backend follow-ups: stripped-plan load check, safer .ptd save and names, unaligned buffers, test fixes, quantized-model test - #4915
Merged
lanluo-nvidia merged 10 commits intoOct 26, 2026
Conversation
github-actions
Bot
added
component: tests
shoumikhin
force-pushed
the
executorch-load-stripped-plans-with-weights
branch
from
October 22, 2026 21:26
920d7f9 to
f99a7bd
Compare
shoumikhin
changed the title
github-actions
Bot
added
documentation
shoumikhin
force-pushed
the
executorch-load-stripped-plans-with-weights
branch
2 times, most recently
from
October 23, 2026 04:23
0f85920 to
f00f2e4
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
executorch-load-stripped-plans-with-weights
branch
2 times, most recently
from
October 26, 2026 00:53
31a1519 to
8d82eb5
Compare
shoumikhin
changed the title
…ghts The load check from pytorch#4848 refuses a TensorRT plan serialized without its weights, because such a plan runs without an error but computes with placeholder weights. It treated any kSTRIPPED_WEIGHTS_SIZE of 0 or more as "weights missing". But for a plan built with STRIP_PLAN, that number is the weight bytes the plan still holds, and it equals the total when TensorRT had nothing to strip. Those plans compute correctly and used to load, including .pte files exported before the check. Now a plan is refused only when it holds fewer weight bytes than its total. The comment and README said -1 means the plan carries its weights. It only means the plan is not marked as stripped. A plain re-save of a stripped plan keeps the mark and is refused. But a stripped plan saved again with EXCLUDE_WEIGHTS cleared and no refit reads -1, so this check cannot catch it, and the old one could not either. Both now say that. New tests: a stripped plan that kept all its weights loads and matches an ordinary engine, and it fails without this fix. A refittable plan saved with EXCLUDE_WEIGHTS is refused. The old check refused it too, so that test guards the behavior rather than proving the fix.
save(file_path="engines.ptd", external_engine_data="engines") wrote the program, then wrote the engines to the same file, and returned normally. Only the engines were left, so there was no program to load. The same happens with another backend's data file, for example "aoti_cuda_blob.ptd" for a program with a CUDA partition, and with "ENGINES.PTD" on a file system that ignores case. Every data file save() writes is <name>.ptd beside the program. So save() now refuses a program path that ends in .ptd, ignoring case and trailing dots and spaces as Windows does, and it does so before it exports anything.
After writing a program's external data, save() logged "point the runtime's data-files at this directory". The loaders want the .ptd file itself: Runtime.load_program(pte, data_path=<directory>) fails with "Failed to load data map", and so does the C++ Module. The log line now lists the full path of every .ptd file it wrote and says to pass that file, or those files, and not the directory.
The deployment and saving guides did not mention external_engine_data=, so a reader learned neither that such a program needs its .ptd at load nor how to pass it. And save() replaces an existing .ptd of the same name, so two programs saved into one directory with the same name share one file, and the first one then fails to load with no warning at save time. Add a short section to the ExecuTorch guide with a save and load example, a note that the .ptd is trusted input like the .pte, and a warning about the shared name. The warning also covers a program with a CUDA partition, which needs its own directory and the CUDA backend's data name. Link the section from the saving guide, and say the same in the save() docstring.
The .ptd name check refused directories, drive prefixes and a trailing .ptd, but let through names that cannot become a file: Windows device names such as NUL or CON, characters Windows does not allow (? * | < > and the double quote), control characters, and names too long for a file system. save() then failed only when it wrote the .ptd, after the .pte was already on disk, and a device name might not fail at all. Its comment also gave a reason for refusing "." and ".." that does not apply, and one message covered every rule, so "engines:v2" was told it had a directory or a drive. Now the name must use only ASCII letters, digits, ".", "-" and "_", start with a letter, a digit or "_", and fit the 255 byte file name limit once .ptd is added. It may not end with .ptd or be a Windows device name. Each kind of problem gets its own message. This also refuses some names that worked on Linux before, such as ones with a space, a "+" or a non-ASCII letter, so a program saved on one system does not fail to save or load on another.
…T_ADDOPTS Two tests run another test in a child pytest process and then check that it left no copy of torch_tensorrt_executorch_runtime in sys.modules. The child inherited PYTEST_ADDOPTS. With "-n 2" in it, the inner test ran in an xdist worker, so the child's own sys.modules never held the module and both checks passed even with the leak fixes removed. "-o addopts=" clears only the ini value, not the variable. Run the child without PYTEST_ADDOPTS, give it a timeout so a hung child fails the test instead of the whole job, and make a failure say which of the two things went wrong.
Three cases reached an allocation failure by filling the GPU with cudaMalloc. On a Jetson the GPU memory is the system RAM, so this took all of the machine's memory and froze it. The failure is now injected through a cudaMalloc wrapper (-Wl,--wrap=cudaMalloc), which takes no memory and leaves the same pending CUDA error as a real out-of-memory. How much activation scratch an empty batch asks for depends on the GPU: some answer zero, others (an aarch64 Jetson board, for example) answer a real size. The two cases that need the zero answer now skip with a reason where they do not get it, so the end-of-suite skip list names them, instead of failing. The empty-batch half of AnUndersizedScratchInstallIsSeenAsRefused moves into its own case so that skip does not hide the first half. The two drain cases skip on a GPU that reads host memory directly, where no input is staged.
…rch runtime Nothing tested a quantized model on the ExecuTorch path. A model quantized with NVIDIA Model Optimizer, compiled with Torch-TensorRT and saved with output_format="executorch" could lose its quantization without anyone noticing, for example if TensorRT kept the weights in float. The new test quantizes a small two-layer model to int8 and, on GPUs with compute capability 8.9 or newer, to fp8. It saves the model as a .pte and checks that: - only TensorRTBackend runs it, besides the host and device copies - the .pte is less than half the size of the same model saved unquantized - the ExecuTorch runtime returns exactly what the compiled module returns Tested on a Jetson AGX Thor with TensorRT 11.3 and 11.4: both cases pass. The size check fails when the model is saved unquantized, and, on TensorRT 11.3, with Model Optimizer's int8 weight-only config, whose weights TensorRT keeps in float there.
… ones TensorRT needs every input and output address to be a multiple of 256 bytes (setTensorAddress documents this) but never checks it. The ExecuTorch delegate passed the caller's pointer straight through, so a device tensor that starts partway into a larger allocation, such as x[1:] of a 1-D tensor, reached the engine as it was. Measured on an H100 with TensorRT 11.4: - an unaligned output, an unaligned in-place input, or an unaligned input to a fused kernel faults with "misaligned address". execute() fails with a log that blames a short buffer or freed storage, and every later CUDA call in the process fails too. - an unaligned input to a matrix multiply returns a wrong answer with no error at all. The delegate already stages host buffers through per-binding device buffers it allocates itself, and cudaMalloc returns 256-byte aligned memory. A buffer at an unaligned address now takes that same path: inputs are copied in before the enqueue, outputs are copied back after it. An input that an output updates in place gets its update copied back as well, whether or not the program elides that output. Aligned buffers are bound directly, as before, with no copy. The first unaligned buffer an engine meets is logged once. ExecuTorch's memory planner places tensors on 16-byte boundaries by default, so a planned buffer can take this path too, and on a device that reads host memory directly so can an unaligned host buffer. The result is correct either way; the cost is one copy per buffer per call, two for a buffer updated in place. Tests: a C++ target of its own, because a fault leaves the CUDA context unusable for the rest of the process, binds an unaligned input, an unaligned output and an unaligned in-place input, the last with its output both passed and elided. Without the fix the input case returns a wrong answer and the output and in-place cases fault; with it every case matches the expected values. A Python test runs a device-resident .pte on x[1:] in float32 and float16 in a child process: without the fix it fails with "misaligned address", with it the output matches the compiled module bit for bit.
shoumikhin
force-pushed
the
executorch-load-stripped-plans-with-weights
branch
from
October 26, 2026 07:17
8d82eb5 to
c7afbce
Compare
shoumikhin
changed the title
The delegate copies any input or output that is not on a 256-byte boundary, because TensorRT needs that and does not check. ExecuTorch's memory planner places tensors on 16-byte boundaries by default, so in an ordinary export a planned delegate buffer can miss the boundary and take that copy on every call. save() now hands to_executorch() a config whose MemoryPlanningPass aligns to the least common multiple of its own alignment and 256. It works on a copy, so the caller's planner keeps its own alignment, and the alignment the caller asked for still holds in the program. A planner that is not a MemoryPlanningPass is left alone. On the export() plus to_executorch() path the caller passes the config, and zero_copy_backend_config() keeps the caller's planner, so the docs say what to pass there. Measured on an H100 with a two-engine block (LayerNorm, Linear, GELU, a multiply left outside TensorRT, Linear, residual), counting the delegate's staging copies per method call: - host inputs and outputs (the default): 4 of 6 bindings before, 0 after - device-resident inputs and outputs: 3 of 6 before, 0 after The planned arenas grow by padding only: 25200 to 25344 bytes and 14000 to 14080 bytes for this model. A caller's buffer off the boundary, such as x[1:], is still copied. Tests: a new test saves that block in both modes and checks that every planned delegate buffer in the .pte starts at a multiple of 256 bytes. Without this change it fails on offsets 8400 and 11200. The save() tests that expected the config to pass through unchanged now check the alignment instead, and that ExecuTorch's shared default planner is not changed in place.
lanluo-nvidia
pushed a commit
that referenced
this pull request
Oct 26, 2026
…eck, safer .ptd save and names, unaligned buffers, test fixes, quantized-model test (#4915)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
kSTRIPPED_WEIGHTS_SIZE, the weight bytes a stripped plan kept, which equals the total when nothing could be stripped.save()can overwrite the program, as insave(file_path="engines.ptd", external_engine_data="engines"). Its log names a directory, not the.ptdfile..ptdnames are accepted: Windows device names likeNUL, characters Windows forbids, and overlong names.-n 2fromPYTEST_ADDOPTS.test_shared_scratch_backendcan freeze a Jetson by filling GPU memory, which is system memory there.x[1:]can give a wrong answer, or fault and break later CUDA calls. TensorRT needs 256-byte aligned addresses and does not check.What this change does
.ptd(any letter case), and log every.ptdpath..,-and_, start with a letter, digit or_, fit 255 bytes with.ptd, and not be a device name. Spaces and+now fail.PYTEST_ADDOPTS, with a timeout.-Wl,--wrap=cudaMalloc, which takes no memory..pteruns only on TensorRT, is under half the float size, and matches the compiled module.save()now plans memory on 256-byte boundaries, so planned GPU buffers need no copy. On a two-engine model, ExecuTorch's default 16-byte planning copies 4 of 6 bindings per call (3 of 6 with GPU-resident inputs and outputs). 256-byte planning copies none.What was tested
Supersedes #4902, #4916, #4917 and #4918.