ggml-backend : copy user inputs before cross-device inputs in compute_splits - #28874
Draft
douyamv wants to merge 1 commit into
Draft
ggml-backend : copy user inputs before cross-device inputs in compute_splits#28874douyamv wants to merge 1 commit into
douyamv wants to merge 1 commit into
Conversation
…_splits With n_copies == 1 (no pipeline parallelism, e.g. whenever tensor overrides are used) the user-input copy synchronizes the whole stream of the split backend. If a cross-device input (the output of the previous split) was processed first, that stream already carries a wait event on the previous device, so the host blocks until the previous GPU has finished its entire graph -- at every split boundary of every decode step. Processing the user inputs first keeps the sync trivial (the stream only holds finished work) and lets the host enqueue the next split immediately. No extra memory, same results. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
github-actions
Bot
added
the
ggml
|
Hi @douyamv, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In
ggml_backend_sched_compute_splits, inputs of a split are copied in graph order. Without pipeline parallelism(
n_copies == 1, which is the case whenever--override-tensoris used sincehas_tensor_overrides()disablesit) the user-input path calls
ggml_backend_synchronize(split_backend). If a cross-device input (the previoussplit's output) was already processed for the same split, that stream carries a wait event on the previous device,
so the synchronize blocks the host until the previous GPU has finished its entire graph — at every split boundary
of every decode step. The CUPTI trace of a 3-GPU layer split showed the host stuck in
cudaStreamSynchronizeforthe full duration of each device's graph before it could enqueue the next one.
Copying the user inputs first (their stream only holds finished work at that point) and the cross-device inputs
afterwards removes the stall with no extra memory and no change in results.
🤖 Generated with Claude Code