Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
347 changes: 159 additions & 188 deletions docs/source/features/checkpoint-loading.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/source/features/sleep-wakeup.md
Original file line number Diff line number Diff line change
Expand Up @@ -355,7 +355,7 @@ Principal implementation locations are:
- Workers: `tensorrt_llm/executor/base_worker.py` and
`tensorrt_llm/executor/ray/gpu_worker.py`
- Allocation tagging: `tensorrt_llm/_torch/pyexecutor/py_executor_creator.py`
and `tensorrt_llm/_torch/pyexecutor/model_loader.py`
and `tensorrt_llm/_torch/models/checkpoints/memory_init.py`
- Python VMM wrapper: `tensorrt_llm/_torch/virtual_memory.py`
- CUDA VMM: `cpp/tensorrt_llm/runtime/virtualMemory.cpp` and
`cpp/include/tensorrt_llm/runtime/virtualMemory.h`
Expand Down
4 changes: 2 additions & 2 deletions docs/source/llm-api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,8 +182,8 @@ The `py_executor.model_engine_creation_seconds` timing includes the total `model
| `draft_checkpoint_preparation_seconds` | Checkpoint preparation time for draft weights loaded as part of the model loader. |
| `draft_weight_population_seconds` | Weight population time for draft weights loaded as part of the model loader. |
| `draft_checkpoint_finalization_seconds` | Checkpoint finalization time for draft weights loaded as part of the model loader. |
| `post_load_processing_seconds` | Time spent in format-specific hooks and model finalization, including post-load weight transformation, quantization and memory cleanup. |
| `total_model_loading_seconds` | Overall model construction and loading interval measured after checkpoint configuration validation. It includes the named phases below. |
| `post_load_processing_seconds` | Time spent in format-specific hooks and model finalization, including post-load weight transformation, quantization, memory cleanup and other work. |
| `total_model_loading_seconds` | Overall model loading interval, from loading and validating the checkpoint configuration to the end of post-load processing. It includes the named phases above. |

`trtllm-serve` exposes the same rank-0 payload in the `startup_metrics` field of the
`GET /server_info` response:
Expand Down
332 changes: 0 additions & 332 deletions docs/source/torch/features/checkpoint_loading.md

This file was deleted.

10 changes: 6 additions & 4 deletions tensorrt_llm/_torch/memory/gpu_memory_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,12 @@

Design notes:
- We deliberately avoid the upstream `setup_gms()` monkey-patch entry
point. TRT-LLM owns the integration policy in `ModelLoader`; this
point. TRT-LLM owns the integration policy in
`models/checkpoints/gms/model_loading.py`; this
backend is a thin adapter over the GMS library's primitive API.
- The protocol has a single point of contact per concern, so when the
upstream GMS Python API drifts, only the methods on `GMSBackend`
need to change — the call sites in `model_loader.py` are stable.
need to change — the call sites in `gms/model_loading.py` are stable.

Operating modes:
- **RW (Read-Write)**: First worker loads weights via the normal
Expand Down Expand Up @@ -120,8 +121,9 @@ class GMSBackend:
This adapter calls the GMS library's stable per-call primitives
(`GMSClientMemoryManager` + the helpers in
`gpu_memory_service.client.torch.allocator` and `.module`) and
composes them into the methods TRT-LLM's `ModelLoader` invokes for
the `LoadFormat.GMS` branch. We intentionally do **not** use the
composes them into the methods that TRT-LLM's GMS loading code
(`models/checkpoints/gms/model_loading.py`) invokes for
`LoadFormat.GMS`. We intentionally do **not** use the
upstream `setup_gms()` monkey-patch — TRT-LLM owns the integration
policy and we keep the dependency surface narrow and explicit.
"""
Expand Down
Loading
Loading