Skip to content

Wrap exceptions in std::exception to verbose output on exception. - #1316

Merged
prusnak merged 1 commit into
ggml-org:masterfrom
ivanstepanovftw:stdexception
May 4, 2023
Merged

Wrap exceptions in std::exception to verbose output on exception.#1316
prusnak merged 1 commit into
ggml-org:masterfrom
ivanstepanovftw:stdexception

Conversation

@ivanstepanovftw

@ivanstepanovftw ivanstepanovftw commented May 4, 2023

Copy link
Copy Markdown
Contributor

I am not getting errors thrown as std::string.

Before:

William Safire will walk us through the nuances of badterminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'

After:

William Safire will walk us through the nuances of badterminate called after throwing an instance of 'std::runtime_error'
  what():  failed to open session/7B/hello.bin: No such file or directory

@prusnak
prusnak merged commit 34d9f22 into ggml-org:master May 4, 2023
This was referenced May 24, 2023
Closed
Merged
@Bearsaerker Bearsaerker mentioned this pull request Mar 12, 2025
Closed
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@ivanstepanovftw
Wrap exceptions in std::exception to verbose output on exception. (gg…
12fd970 
…ml-org#1316)
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@ivanstepanovftw
Wrap exceptions in std::exception to verbose output on exception. (gg…
d9e8e1d 
…ml-org#1316)
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@ivanstepanovftw
Wrap exceptions in std::exception to verbose output on exception. (gg…
3eb59f1 
…ml-org#1316)
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
@ivanstepanovftw
Wrap exceptions in std::exception to verbose output on exception. (gg…
3ee7129 
…ml-org#1316)
This was referenced Nov 7, 2026
Merged
Merged
bong-water-water-bong added a commit to 1bit-MONSTER/llama.cpp that referenced this pull request Nov 7, 2026
@bong-water-water-bong
hrx: do not pack a device-resident weight that another node gathers n…
61946e7 
…atively (engine ggml-org#1316)

A tied token embedding is gathered by GET_ROWS and contracted by the output MUL_MAT. The contraction asks for a packed row
layout, so one value is read in two layouts by one command program - and a program holds one binding per value
(CommandProgramBindings::find(ValueId)), so the executor refuses it: "weight value 0 has conflicting resident layout
requests", llama_decode ret = -3. That is why moving the input layer onto the device
(LLAMA_INPUT_LAYER_ON_DEV=1, src/llama-model.cpp:1372) has never worked for a tied embedding, and why the input layer
stays on the CPU and costs one node plus one split per graph.

The dispatch registry now declines a match that requests a packed layout of a weight another node gathers natively, so the
next registration for the same root op - the native-format one - takes the node and both consumers share one resident copy.

Scoped to device-resident weights: the conflict needs both readers to want a resident binding, which only happens when the
weight is already on the device. With the weight in a host buffer - the default, input layer on the CPU - the gate cannot
fire, so the shipped path is structurally unchanged rather than unchanged in one box state (measured: default path 309.5
t/s median with the gate, 309.6 without, three repeats each).

GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0 restores the previous behaviour and is the A/B control.

Measured at 0.6B, -dev HRX0 -ngl 99 -c 512, no-SVM trio: LLAMA_INPUT_LAYER_ON_DEV=1 with GGML_HRX_STRICT=1 now exits 0 with no
node off HRX (3/3 runs; the pin aborts with ret=-3), while the same arm with the gate off still refuses the program (3/3).
bong-water-water-bong added a commit to 1bit-MONSTER/llama.cpp that referenced this pull request Nov 7, 2026
@bong-water-water-bong agent
hrx: do not pack a device-resident weight that another node gathers n…
a85a6f3 
…atively (engine ggml-org#1316)

A tied token embedding is gathered by GET_ROWS and contracted by the output MUL_MAT. The contraction asks for a packed row
layout, so one value is read in two layouts by one command program - and a program holds one binding per value
(CommandProgramBindings::find(ValueId)), so the executor refuses it: "weight value 0 has conflicting resident layout
requests", llama_decode ret = -3. That is why moving the input layer onto the device
(LLAMA_INPUT_LAYER_ON_DEV=1, src/llama-model.cpp:1372) has never worked for a tied embedding, and why the input layer
stays on the CPU and costs one node plus one split per graph.

The dispatch registry now declines a match that requests a packed layout of a weight another node gathers natively, so the
next registration for the same root op - the native-format one - takes the node and both consumers share one resident copy.

Scoped to device-resident weights: the conflict needs both readers to want a resident binding, which only happens when the
weight is already on the device. With the weight in a host buffer - the default, input layer on the CPU - the gate cannot
fire, so the shipped path is structurally unchanged rather than unchanged in one box state (measured: default path 309.5
t/s median with the gate, 309.6 without, three repeats each).

GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0 restores the previous behaviour and is the A/B control.

Measured at 0.6B, -dev HRX0 -ngl 99 -c 512, no-SVM trio: LLAMA_INPUT_LAYER_ON_DEV=1 with GGML_HRX_STRICT=1 now exits 0 with no
node off HRX (3/3 runs; the pin aborts with ret=-3), while the same arm with the gate off still refuses the program (3/3).
bong-water-water-bong added a commit to 1bit-MONSTER/llama.cpp that referenced this pull request Nov 7, 2026
@bong-water-water-bong
hrx: do not pack a device-resident weight that another node gathers n…
67d8c42 
…atively (engine ggml-org#1316) (#233)

A tied token embedding is gathered by GET_ROWS and contracted by the output MUL_MAT. The contraction asks for a packed row
layout, so one value is read in two layouts by one command program - and a program holds one binding per value
(CommandProgramBindings::find(ValueId)), so the executor refuses it: "weight value 0 has conflicting resident layout
requests", llama_decode ret = -3. That is why moving the input layer onto the device
(LLAMA_INPUT_LAYER_ON_DEV=1, src/llama-model.cpp:1372) has never worked for a tied embedding, and why the input layer
stays on the CPU and costs one node plus one split per graph.

The dispatch registry now declines a match that requests a packed layout of a weight another node gathers natively, so the
next registration for the same root op - the native-format one - takes the node and both consumers share one resident copy.

Scoped to device-resident weights: the conflict needs both readers to want a resident binding, which only happens when the
weight is already on the device. With the weight in a host buffer - the default, input layer on the CPU - the gate cannot
fire, so the shipped path is structurally unchanged rather than unchanged in one box state (measured: default path 309.5
t/s median with the gate, 309.6 without, three repeats each).

GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0 restores the previous behaviour and is the A/B control.

Measured at 0.6B, -dev HRX0 -ngl 99 -c 512, no-SVM trio: LLAMA_INPUT_LAYER_ON_DEV=1 with GGML_HRX_STRICT=1 now exits 0 with no
node off HRX (3/3 runs; the pin aborts with ret=-3), while the same arm with the gate off still refuses the program (3/3).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ivanstepanovftw @ggerganov @prusnak