Notes

Note: This post was originally written in 2021, but I have since updated it to reflect the latest changes in HuggingFace Accelerate (last update November 2025 using accelerate==1.11.0). For a grad course that recently concluded, the course project required me to train and evaluate a large number of models. Our school’s local SLURM cluster has new GPUs that support fp16, which meant I could take advantage of PyTorch’s Automatic Mixed Precision (AMP) training. And honestly, there is no reason not to use it: we get reduced memory usage, faster training, and all of this without virtually any loss in performance. ...

We recently got an email from our IT department that our workstation OSes will be getting upgraded from Ubuntu 18.04 MATE to Ubuntu 20.04 GNOME. As much as I love MATE and how lightweight it is (LinuxScoop makes wonderful OS overview videos), I also like the “visuals” of GNOME. My personal laptop already runs Ubuntu 20.04 GNOME, so I am excited to have it on my lab workstation as well. However, this OS upgrade also means that we have to backup our workstations since the drives will be wiped. Our research group has a generous storage space allocation on Compute Canada’s Cedar, so storage is not a big issue. The problem is: Cedar’s long-term storage space is a “tape-based backup system”, so there is a strict limit on the number of files we can store there. Therefore, the best strategy is to create tar archives of our data and store those on Cedar. ...