I usually rely on running a wc -l (line count) to do a sanity check on the number of rows in a file (e.g., dataset metadata). Yesterday, I lost an unhealthy amount of time debugging a weird discrepancy between the wc -l output on a CSV file and the number of rows after I read it into a pandas DataFrame. And I learned something new about it. I guess I was one of 10,000 lucky people yesterday. ...
For years, my Python workflow relied on conda for environment and package management, pip for package management, black for auto-formatting, and isort for sorting imports. In places where I couldn’t use conda, such as Digital Research Alliance of Canada (FKA Compute Canada) clusters or own school’s own SLURM cluster, I would use virtualenv/venv to manage my environments and ocassionally, conda-pack when I absolutely needed to use the exact conda environment on a cluster. ...