Skip to content

[Superseded: original Draft design 41] Add an independent eager attention backend - #1122

Closed
jiapingW wants to merge 1 commit into
design/40-dynamic-rotaryfrom
design/41-eager-attention-backend
Closed

jiapingW wants to merge 1 commit into
design/40-dynamic-rotaryfrom
design/41-eager-attention-backend

Conversation

@jiapingW

@jiapingW jiapingW commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Adds an independent eager softmax-attention backend under layers/attention/backends/. It consumes prepared Q/K/V tensors and an explicit mask, scale, dropout probability, and training flag. It owns no projections, architecture settings, learned parameters, or causal/window policy.

The backend supports MHA/grouped KV heads, unequal query/key lengths, and distinct query/value widths. Boolean masks use True for visibility; floating masks are additive, with negative infinity blocking a key. Fully blocked rows return zero with zero gradients instead of NaNs. Finite negative biases retain their additive meaning. Invalid tensor/mask layouts fail before attention computation.

Validation on H20-wjp:

  • 7 backend tests passed. CPU FP32/BF16 and CUDA FP32/FP16/BF16 outputs and Q/K/V/bias gradients match HF Qwen3 eager attention exactly in the tested ordinary-row cases, including training dropout, broadcast masks, noncontiguous inputs, and grouped heads.
  • CPU/GPU fully blocked rows agree with the PyTorch SDPA math reference (rtol 1e-5, atol 1e-6); blocked outputs and their gradients are exactly zero. Additional tests cover block/context isolation, empty query/key lengths, and early validation.
  • 231 configuration tests and 36 existing component/HF-loading/backbone tests passed. The parent PR's 27 rotary tests and both PRs' pre-commit checks also passed.

Uses the existing Transformers 5.12.1 environment; the project's pinned 5.19.0 environment was not exercised. This supplies the numerical backend for subsequent attention components; it does not yet implement/register GQA projections, build algorithm masks, or migrate model entry points. No serving/performance claim is made. Stacked on #1121; one commit.

Historical PR for original Draft design 41. Superseded by current Draft design 06/21, #1124. Its code is included in that PR; this historical PR remains closed.

@jiapingW
jiapingW force-pushed the design/40-dynamic-rotary branch from 226e9e6 to bbb4c57 Compare November 8, 2026 16:21
@jiapingW
jiapingW force-pushed the design/41-eager-attention-backend branch 3 times, most recently from dd00e14 to c8f1731 Compare November 8, 2026 17:45
@jiapingW jiapingW closed this Nov 8, 2026
@jiapingW jiapingW changed the title [Draft design 41] Add an independent eager attention backend [Superseded: original Draft design 41] Add an independent eager attention backend Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant