Repository navigation

Fix float16 overflow in clip_grad_norm - #4626

Merged

zcbenz merged 3 commits into

ml-explore:mainfrom

ishtihoss:clip-grad-norm-fp16-overflow-zeroes-grads

Oct 7, 2026

Merged

Fix float16 overflow in clip_grad_norm#4626
zcbenz merged 3 commits into
ml-explore:mainfrom
ishtihoss:clip-grad-norm-fp16-overflow-zeroes-grads

Commits on Oct 4, 2026
- clip_grad_norm squared and summed gradients in their own dtype, so any fp16 gradient above ~256 overflowed to inf; making the total norm inf and the normalizer zero; so literally every gradient sil…
  ishtihoss
  committed
  95b9228
Commits on Oct 5, 2026
- fixed the failing tests
  ishtihoss
  committed
  43426ca
- Merge branch 'main' into clip-grad-norm-fp16-overflow-zeroes-grads
  ishtihoss
  authored
  251ea08