I usually rely on running a wc -l (line count) to do a sanity check on the number of rows in a file (e.g., dataset metadata). Yesterday, I lost an unhealthy amount of time debugging a weird discrepancy between the wc -l output on a CSV file and the number of rows after I read it into a pandas DataFrame. And I learned something new about it. I guess I was one of 10,000 lucky people yesterday. ...
I recently wrote about using fastfetch as a part of my MOTD (Message of the Day) in this post. I like it a lot and wanted it to be available on my lab workstation as well, which runs Ubuntu 22.04. However, I do not have sudo access on this machine, which means I cannot just sudo apt install fastfetch like I did on my home computer. So, I did the next best thing (relevant xkcd?): compile it from source. The process was fairly straightforward, but I had never done it before, so I thought I might as well document it for future reference. ...
I spend a lot of time ssh-ing into workstations, and I always found the default login banner to be functional albeit a bit bland. This, coupled with how many fastfetch/neofetch posts I see on reddit, I decided to implement it on my own home workstation. This adds a bit of visual flair as well as useful system stats to the login prompt. I originally wanted to use neofetch but it has been archived and is no longer maintained. fastfetch, on the other hand, is almost a drop-in replacement and is actively maintained, so I decided to use that instead. ...
I used to rely on DuckDNS to make my workstations accessible over the internet. However, I found that my auth logs were getting spammed with failed login attempts. Click to see a sample of the auth logs $ journalctl -f Feb 27 21:31:54 prometheus sshd[28079]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:31:54 prometheus sshd[28079]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:31:55 prometheus sshd[28079]: Failed password for invalid user user from 185.196.220.81 port 51248 ssh2 Feb 27 21:31:55 prometheus sshd[28077]: Failed password for root from 218.92.0.243 port 54595 ssh2 Feb 27 21:31:56 prometheus sshd[28079]: Received disconnect from 185.196.220.81 port 51248:11: end [preauth] Feb 27 21:31:56 prometheus sshd[28079]: Disconnected from invalid user user 185.196.220.81 port 51248 [preauth] Feb 27 21:31:58 prometheus sshd[28083]: Invalid user user from 185.196.220.81 port 37162 Feb 27 21:31:58 prometheus sshd[28083]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:31:58 prometheus sshd[28083]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:31:59 prometheus sshd[28077]: Failed password for root from 218.92.0.243 port 54595 ssh2 Feb 27 21:32:00 prometheus sshd[28083]: Failed password for invalid user user from 185.196.220.81 port 37162 ssh2 Feb 27 21:32:02 prometheus sshd[28077]: Failed password for root from 218.92.0.243 port 54595 ssh2 Feb 27 21:32:03 prometheus sshd[28083]: Received disconnect from 185.196.220.81 port 37162:11: end [preauth] Feb 27 21:32:03 prometheus sshd[28083]: Disconnected from invalid user user 185.196.220.81 port 37162 [preauth] Feb 27 21:32:03 prometheus sshd[28077]: Received disconnect from 218.92.0.243 port 54595:11: [preauth] Feb 27 21:32:03 prometheus sshd[28077]: Disconnected from authenticating user root 218.92.0.243 port 54595 [preauth] Feb 27 21:32:03 prometheus sshd[28077]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.243 user=root Feb 27 21:32:04 prometheus sshd[28105]: Invalid user Admin from 185.196.220.81 port 37176 Feb 27 21:32:04 prometheus sshd[28105]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:32:04 prometheus sshd[28105]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:32:06 prometheus sshd[28105]: Failed password for invalid user Admin from 185.196.220.81 port 37176 ssh2 Feb 27 21:32:06 prometheus sshd[28105]: Received disconnect from 185.196.220.81 port 37176:11: end [preauth] Feb 27 21:32:06 prometheus sshd[28105]: Disconnected from invalid user Admin 185.196.220.81 port 37176 [preauth] Feb 27 21:32:07 prometheus sshd[28110]: Invalid user admin from 185.196.220.81 port 52236 Feb 27 21:32:07 prometheus sshd[28110]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:32:07 prometheus sshd[28110]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:32:09 prometheus sshd[28110]: Failed password for invalid user admin from 185.196.220.81 port 52236 ssh2 Feb 27 21:32:10 prometheus sshd[28110]: Received disconnect from 185.196.220.81 port 52236:11: end [preauth] Feb 27 21:32:10 prometheus sshd[28110]: Disconnected from invalid user admin 185.196.220.81 port 52236 [preauth] Feb 27 21:32:11 prometheus sshd[28122]: Invalid user admin from 185.196.220.81 port 52246 Feb 27 21:32:11 prometheus sshd[28122]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:32:11 prometheus sshd[28122]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:32:13 prometheus sshd[28122]: Failed password for invalid user admin from 185.196.220.81 port 52246 ssh2 Feb 27 21:32:14 prometheus sshd[28122]: Received disconnect from 185.196.220.81 port 52246:11: end [preauth] Feb 27 21:32:14 prometheus sshd[28122]: Disconnected from invalid user admin 185.196.220.81 port 52246 [preauth] Feb 27 21:32:15 prometheus sshd[28146]: Invalid user user from 185.196.220.81 port 52252 Feb 27 21:32:15 prometheus sshd[28146]: pam_unix(sshd:auth): check pass; user unknown Feb 27 21:32:15 prometheus sshd[28146]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.196.220.81 Feb 27 21:32:17 prometheus sshd[28146]: Failed password for invalid user user from 185.196.220.81 port 52252 ssh2 Feb 27 21:32:19 prometheus sshd[28146]: Received disconnect from 185.196.220.81 port 52252:11: end [preauth] Feb 27 21:32:19 prometheus sshd[28146]: Disconnected from invalid user user 185.196.220.81 port 52252 [preauth] Feb 27 21:32:35 prometheus sshd[28310]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.230 user=root Feb 27 21:32:36 prometheus sshd[28310]: Failed password for root from 218.92.0.230 port 30964 ssh2 Feb 27 21:32:38 prometheus sshd[28310]: Failed password for root from 218.92.0.230 port 30964 ssh2 Feb 27 21:32:43 prometheus sshd[28310]: Failed password for root from 218.92.0.230 port 30964 ssh2 Feb 27 21:32:43 prometheus sshd[28310]: Received disconnect from 218.92.0.230 port 30964:11: [preauth] Feb 27 21:32:43 prometheus sshd[28310]: Disconnected from authenticating user root 218.92.0.230 port 30964 [preauth] Feb 27 21:32:43 prometheus sshd[28310]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.230 user=root Unsurprisingly, this is a common issue. So much so that there are databases of such IP addresses. For example, the last IP address from the logs above has been reported 100k+ times.. When I brought this up to my colleague Ben, he suggested using fail2ban. Looking into more options, I found endlessh to be a good solution. So here is what I did: ...
As deep learning {datasets, models, scale of experiments} grow, so do the storage requirements, and we increasingly find ourselves running out of space on our SSDs. I recently added a new 8TB HDD to my workstation to act as a new scratch volume. While adding a disk would be a standard “sysadmin” work, I found the process of handling permissions for a shared research group on a domain-connected Linux machine to be a bit more involved than I expected. So, for the sake of my own documentation and on the off chance that someone else might find it useful, I’ll document the process here. Since I am using Ubuntu 22.04 on this workstation, I used the Ubuntu instrcutions as a starting point and modified them as needed. ...