You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Setting all parameters to FP16 in the encoder leads to overflow in the LayerNorm layers. We suggest you to enable FP16 mode while forcing the LayerNorm layers to FP32 precision. This approach has been verified by us. I hope this addresses your question. Please feel free to reach out if you have additional concerns.
Activity
zhuoyang20 commentedon Oct 24, 2024
Hi,
Setting all parameters to FP16 in the encoder leads to overflow in the LayerNorm layers. We suggest you to enable FP16 mode while forcing the LayerNorm layers to FP32 precision. This approach has been verified by us. I hope this addresses your question. Please feel free to reach out if you have additional concerns.
Best,
Zhuoyang