11. Troubleshooting and cheat sheet¶
11.1 Errors and what they mean¶
Symptom |
Likely cause |
Fix |
|---|---|---|
|
Wrong key, username or port, or the key is not installed yet |
Check |
Host identification warning |
A login node you have not met, or a real problem |
Read your cluster's guidance before accepting |
|
Trailing backslash in a Windows path |
Remove the backslash |
|
|
|
|
Too much memory on a shared node |
Use |
|
Shell not set up for conda |
|
|
The environment was never created, often as an earlier step failed |
Scroll up and fix the first error |
|
|
|
|
The job is using a different Python |
Call your Python by its full path |
|
Only the installer is loaded |
|
Downloads fail in a job |
Compute nodes may have restricted internet |
Download on the login node first, and build offline |
|
Read-only after copying |
|
|
Windows line endings |
|
No log file appears |
The |
|
State |
Hit the time limit |
More |
State |
Hit the memory limit |
More |
Stuck on |
Busy, or you asked for too much |
|
Files vanished from scratch |
Purged after no access |
Keep copies elsewhere |
Pasted lines ran in an odd order |
Multi-line paste in a terminal |
Paste one line at a time and check |
The general method: read the first error rather than the last, as later errors are often consequences. Then search for its exact text.
11.2 Cheat sheet¶
## cluster: daily use
squeue -u $USER # what is running
sbatch job.sh # submit
sbatch --array=0-9 job.sh args # submit an array
scancel <jobid> # cancel
tail -20 "$(ls -t logs/*.out | head -1)" # newest log
sacct -X -P -S 2026-01-01 -u $USER -o JobID,JobName,Elapsed,State # history
srun --cpus-per-task=1 --mem=4G --time=00:15:00 --pty bash # interactive
module avail; module load <name>; module list # software
quota -s; du -sh ~/project # storage
tar czf out.tgz folder; tar xzf out.tgz # pack and unpack
## laptop
ssh kelvin # log in
scp file kelvin:~/project/ # upload
scp kelvin:~/project/file "D:\My Work" # download
11.3 References¶
Amdahl, G.M. (1967) 'Validity of the single processor approach to achieving large scale computing capabilities', AFIPS Spring Joint Computer Conference, pp. 483-485. https://doi.org/10.1145/1465482.1465560
Klabnik, S. and Nichols, C. (n.d.) The Rust Programming Language. https://doc.rust-lang.org/book/
Kurtzer, G.M., Sochat, V. and Bauer, M.W. (2017) 'Singularity: scientific containers for mobility of compute', PLoS ONE, 12(5), e0177459. https://doi.org/10.1371/journal.pone.0177459
McNabb, R. (n.d.) Introduction to high performance computing. https://iamdonovan.github.io/teaching/hpc-intro/index.html
Merkel, D. (2014) 'Docker: lightweight Linux containers for consistent development and deployment', Linux Journal, 2014(239).
NI-HPC (n.d.) NI-HPC documentation. https://ni-hpc.github.io/nihpc-documentation
OpenSSH (n.d.) OpenSSH manual pages. https://www.openssh.com/manual.html
Raicu, I., Foster, I.T. and Zhao, Y. (2008) 'Many-task computing for grids and supercomputers', Workshop on Many-Task Computing on Grids and Supercomputers, pp. 1-11. https://doi.org/10.1109/MTAGS.2008.4777912
SchedMD (n.d.) Slurm job array support. https://slurm.schedmd.com/job_array.html
Shotts, W. (2019) The Linux Command Line. 5th internet edn. https://linuxcommand.org/tlcl.php
Wilson, G., Aruliah, D.A., Brown, C.T., et al. (2014) 'Best practices for scientific computing', PLoS Biology, 12(1), e1001745. https://doi.org/10.1371/journal.pbio.1001745
Yoo, A.B., Jette, M.A. and Grondona, M. (2003) 'SLURM: Simple Linux Utility for Resource Management', in Job Scheduling Strategies for Parallel Processing. Springer, pp. 44-60. https://doi.org/10.1007/10968987_3