Training

Note: This post was originally written in 2021, but I have since updated it to reflect the latest changes in HuggingFace Accelerate (last update November 2025 using accelerate==1.11.0). For a grad course that recently concluded, the course project required me to train and evaluate a large number of models. Our school’s local SLURM cluster has new GPUs that support fp16, which meant I could take advantage of PyTorch’s Automatic Mixed Precision (AMP) training. And honestly, there is no reason not to use it: we get reduced memory usage, faster training, and all of this without virtually any loss in performance. ...