#mixed-precision

Loss becomes NaN after switching to fp16 mixed precision

We enabled fp16 autocast to speed up training and the loss turned NaN within 300 steps. A custom attention score and a loss computed in half precision overflowed fp16's range, and we had not used a gradient scaler. bf16 on our GPUs, fp32 for the sensitive ops and a NaN guard fixed it.