As deep learning {datasets, models, scale of experiments} grow, so do the storage requirements, and we increasingly find ourselves running out of space on our SSDs. I recently added a new 8TB HDD to my workstation to act as a new scratch volume. While adding a disk would be a standard “sysadmin” work, I found the process of handling permissions for a shared research group on a domain-connected Linux machine to be a bit more involved than I expected. So, for the sake of my own documentation and on the off chance that someone else might find it useful, I’ll document the process here. Since I am using Ubuntu 22.04 on this workstation, I used the Ubuntu instrcutions as a starting point and modified them as needed.

Important: This guide assumes that you are using a domain-connected Linux machine and that you have sudo access to the machine.

Step 1: Identify the new disk

Assuming your SATA disk is connected, we first need to identify it and fine the device name of the physical disk. We can do this by running the following command:

 1$ lsblk              
 2NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
 3sda      8:0    0   3.7T  0 disk 
 4└─sda1   8:1    0   3.7T  0 part /local-scratch2
 5sdb      8:16   0 931.5G  0 disk 
 6├─sdb1   8:17   0     1M  0 part 
 7├─sdb2   8:18   0   488M  0 part /boot/efi
 8├─sdb3   8:19   0   977M  0 part /boot
 9├─sdb4   8:20   0  30.5G  0 part [SWAP]
10├─sdb5   8:21   0 119.2G  0 part /
11└─sdb6   8:22   0 780.4G  0 part /local-scratch
12sdc      8:32   0   7.3T  0 disk

As you can see, in my case, sda and sdb were already in use by my existing disks, so my new (7.3T) disk was identified as sdc.

Step 2: Partition the disk

For large drives (i.e., over 2TB), fdisk will not be ideal because of the MBR limitations. I do not know much about it so I will just reference what seems to be a good guide/explanation. Instead, we will use parted to create a GPT partition table.

$ sudo parted /dev/sdc
GNU Parted 3.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.

This will return a parted prompt where we can:

  • create a GPT partition table
  • create a single primary partition that spans the entire disk
  • set our unit to TB
  • specify the file system type (e.g., ext4, xfs, etc.)
  • print the partition table to verify our settings
  • quit parted
(parted) mkpart
Partition name?  []? primary                                              
File system type?  [ext2]? xfs                                            
Start? 0                                                                  
End? 8

Setting the start to 0 and end to 8 will create a single primary partition that spans the entire disk, since our disk is 8TB. As for the file system type, I have only used ext4 and btrfs in the past, but researching online seemed to suggest that xfs was a good choice for larger disks, so I went with that.

Next, verify the partition table and then quit parted. Remember, the changes are not saved until parted is quit.

(parted) print

Model: ATA ST8000VN004-2M21 (scsi)
Disk /dev/sdc: 8.00TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      0.00TB  8.00TB  8.00TB  xfs          primary

(parted) quit
Information: You may need to update /etc/fstab.

Let us run lsblk once again to verify that our partition is now visible (sdc1):

 1$ lsblk              
 2NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
 3sda      8:0    0   3.7T  0 disk 
 4└─sda1   8:1    0   3.7T  0 part /local-scratch2
 5sdb      8:16   0 931.5G  0 disk 
 6├─sdb1   8:17   0     1M  0 part 
 7├─sdb2   8:18   0   488M  0 part /boot/efi
 8├─sdb3   8:19   0   977M  0 part /boot
 9├─sdb4   8:20   0  30.5G  0 part [SWAP]
10├─sdb5   8:21   0 119.2G  0 part /
11└─sdb6   8:22   0 780.4G  0 part /local-scratch
12sdc      8:32   0   7.3T  0 disk 
13└─sdc1   8:33   0   7.3T  0 part

Step 3: Format the partition

With the partition created and verified at /dev/sdc1, we can now format it. Again, I went with XFS since it was recommended for larger disks, but you can use any file system you prefer.

$ sudo mkfs -t xfs /dev/sdc1
meta-data=/dev/sdc1              isize=512    agcount=8, agsize=268435455 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=1953506304, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Step 4: Mount the partition

Next, we need to mount the partition. For this, first let’s create a mount point and mount the drive there. Because /local-scratch and /local-scratch2 were already in use, I named this /local-scratch3.

$ cd /
$ sudo mkdir /local-scratch3
$ sudo mount /dev/sdc1 /local-scratch3

This should make the partition available at /local-scratch3.

Step 5: The “tricky” part: Shared group permissions

This is where things got a bit tricky. I wanted to make this drive readable/writable for the “domain users” group, which means we need to set the permissions to something like “domain\users”, but that throws an error. I figured out that we can do this by getting the numeric ID (GID aka group ID) of the user/group and then enabling the permissions for that group ID.

$ id kabhishe
uid=123456(kabhishe) gid=8088(domain users) ...

So, in our case, the gid for “domain users” is 8088. We can now simply apply the permissions for this group ID:

# First, we change the group ownership of the directory to the group ID.
# Important: notice the + sign.
$ sudo chgrp +8088 /local-scratch3

# Next, we grant write access by using appropriate `chmod` flags (g, w; read is enabled by default).
$ sudo chmod g+w /local-scratch3

# Finally, we add a `+t` flag to the directory. From the Ubuntu guide: "# From the guide above: "The last "chmod +t" adds the sticky bit, so that people can only delete their own files and sub-directories in a directory, even if they have write permissions to it (see man chmod)."
$ sudo chmod +t /local-scratch3

[Optional] Step 6: Update /etc/fstab

We need to update /etc/fstab to make the partition mount at boot. This is optional but almost always necessary unless you want to keep mounting it manually every time you reboot.

First, let’s get the UUID of the partition:

$ sudo blkid /dev/sdc1
/dev/sdc1: UUID="12345678-1234-1234-1234-123456789012" TYPE="xfs"

Next, let’s add this partition to /etc/fstab:

$ sudo nano /etc/fstab

Add the following line to the end of the file:

UUID=12345678-1234-1234-1234-123456789012 /local-scratch3 xfs defaults 0 2

Remember, if you do not update /etc/fstab, the partition will not mount at boot and you will need to mount it manually every time you reboot by doing this:

$ sudo mount /dev/sdc1 /local-scratch3

Extra Utils: Getting SSD and HDD info

To get tge details of your HDD, run this (I ran these commands later when only one HDD was connected):

$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST8000VN004-2M21 Rev: SC60
  Type:   Direct-Access                    ANSI  SCSI revision: 05

To get the details of your NVMe SSD(s), run this (needs sudo):

$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S64ANJ0RA28878W      Samsung SSD 980 1TB                      1         617.08  GB /   1.00  TB    512   B +  0 B   1B4QFXO7