Skip to content

[attention] Make the FlexAttention mask type configurable - #5

Open
aws-arashj wants to merge 2 commits into
mainfrom
attention/flex-mask-type
Open

aws-arashj wants to merge 2 commits into
mainfrom
attention/flex-mask-type

Conversation

@aws-arashj

@aws-arashj aws-arashj commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

FlexAttention always builds a block-causal mask, which is causal within each packed document. This PR adds FlexInnerAttention.Config.mask_type so a model can choose plain "causal" instead, matching the transformers backend's attn_mask_type.

  • "block_causal" is the default and keeps today's behavior.
  • "causal" doesn't depend on positions, so the mask is the same for every batch. It's only correct when each sequence holds one document.

A new helper, get_flex_causal_mask_mods(positions, config), returns the selected mask_mods. The decoder, GPT-OSS, and Muse Glimmer mask builders use it instead of composing the masks inline. Masks are unchanged for every existing config.

Test plan

  • New tests/unit_tests/cpu/test_flex_mask_type.py (2 tests, 9 subtests, all passing):
    • mask_mods. On packed and single-document positions, block_causal is causal ∧ same-document and causal is causal. The two agree on single-document sequences.
    • Mask builders. For the decoder, GPT-OSS, and Muse Glimmer, every mask returned with "causal" equals the lower-triangular block mask, and the default block_causal keeps fewer blocks. That covers both the full and sliding-window masks. It fails if either model's builder ignores mask_type.
  • test_flex_attention.py, test_gpt_oss_moe.py, test_muse_glimmer.py, and test_varlen_attention.py: 13 passed, the same as on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant