Storage

As deep learning {datasets, models, scale of experiments} grow, so do the storage requirements, and we increasingly find ourselves running out of space on our SSDs. I recently added a new 8TB HDD to my workstation to act as a new scratch volume. While adding a disk would be a standard “sysadmin” work, I found the process of handling permissions for a shared research group on a domain-connected Linux machine to be a bit more involved than I expected. So, for the sake of my own documentation and on the off chance that someone else might find it useful, I’ll document the process here. Since I am using Ubuntu 22.04 on this workstation, I used the Ubuntu instrcutions as a starting point and modified them as needed. ...

We recently got an email from our IT department that our workstation OSes will be getting upgraded from Ubuntu 18.04 MATE to Ubuntu 20.04 GNOME. As much as I love MATE and how lightweight it is (LinuxScoop makes wonderful OS overview videos), I also like the “visuals” of GNOME. My personal laptop already runs Ubuntu 20.04 GNOME, so I am excited to have it on my lab workstation as well. However, this OS upgrade also means that we have to backup our workstations since the drives will be wiped. Our research group has a generous storage space allocation on Compute Canada’s Cedar, so storage is not a big issue. The problem is: Cedar’s long-term storage space is a “tape-based backup system”, so there is a strict limit on the number of files we can store there. Therefore, the best strategy is to create tar archives of our data and store those on Cedar. ...