10. Other clusters, the cloud and Linux basics¶
Nearly everything in this guide carries over. A new system mostly changes the names, the storage rules and who pays. This chapter is a map to help you adapt, and each system's own documentation has the final word.
10.1 A checklist for any new system¶
-
Who can use it, and how do I apply?
-
How do I log in: which address, port, keys and multi-factor steps?
-
Which storage is backed up, which is fast, and how much do I have?
-
Which scheduler does it use, and which partitions or queues?
-
How is software provided: modules, containers, or my own installs?
-
What are the limits on time, memory and number of jobs?
-
How should I acknowledge it in publications?
Then run Exercise 1 of this guide. If that works, the rest will follow.
10.2 Other schedulers¶
Slurm is common but not universal. The ideas match, while the words differ.
Task |
Slurm |
PBS / Torque |
Grid Engine |
LSF |
|---|---|---|---|---|
Submit |
|
|
|
|
See the queue |
|
|
|
|
Cancel |
|
|
|
|
Script directive |
|
|
|
|
Array task number |
|
|
|
|
10.3 Cluster or cloud?¶
Shared cluster |
Cloud |
|
|---|---|---|
Cost |
Usually free or allocated to researchers |
Paid by use, so a forgotten machine keeps charging |
Waiting |
A queue |
Usually starts at once |
Control |
Fixed system; no administrator rights |
You control the machine, and so must secure and maintain it |
Best for |
Large batches, shared storage |
Special hardware, short bursts, web services |
A cloud virtual machine feels familiar: you create it, log in with SSH using the same kind of key, and run commands on Linux. The usual pattern is:
-
Set a budget alert before you start anything.
-
Create the smallest machine that does the job, and allow SSH only from your own address where you can.
-
Keep data in object storage, such as S3-style buckets, as a machine's own disk may be deleted with it.
-
Run long work inside
tmux, described below, so that it survives a dropped connection. -
Stop or delete the machine when you finish. Check that its disks and addresses are gone too, as they can keep charging.
For many tasks, services such as AWS Batch, Azure Batch and Google Cloud Batch offer something like a job array, and some providers can build a Slurm cluster for you. Containers from the software chapter move well between all of these.
10.4 Linux basics for looking after your own work¶
10.4.1 Permissions¶
Every file has an owner and a group, with read (r), write (w) and execute (x) rights for each.
## any Linux system
## Shows, for example, -rwxr-x--- : owner all, group read and run, others nothing
ls -l run.sh
## Let yourself run a script
chmod u+x run.sh
## Keep a private SSH key readable only by you. SSH refuses keys left more open
chmod 600 ~/.ssh/my-key
10.4.2 Space¶
## any Linux system
## How full are the disks?
df -h
## What is using the space here, biggest last?
du -sh * | sort -h
Many clusters have their own command for your quota. Check its documentation.
10.4.3 Processes¶
## any Linux system
## A live view of what is running. Press q to leave
top
## Just your own programs
ps -u $USER
## Stop a process politely, using its number from ps
kill <process id>
On a cluster, stop jobs with scancel, not kill.
10.4.4 Sessions that survive a disconnection¶
tmux keeps a terminal running on the remote machine after you leave.
## any Linux system
tmux new -s work
## Do your work, then press Ctrl-b, then d, to leave it running
## Come back later
tmux attach -t work
On a cluster, use it for light work on login nodes. Heavy work still belongs in jobs.
10.4.5 Scheduled tasks¶
cron runs commands at set times on a machine you manage. Edit your list with crontab -e. This line runs a backup at 02:00 every day:
0 2 * * * /home/me/backup.sh >> /home/me/backup.log 2>&1
Check whether your cluster allows it before relying on it.
10.4.6 Installing system software¶
On your own machine or cloud server, you have administrator rights through sudo:
## Debian or Ubuntu
sudo apt update && sudo apt install tmux
## Red Hat, Rocky or Fedora
sudo dnf install tmux
On a shared cluster you will not have sudo. Use modules, conda or containers.
10.4.7 Keeping your own server safe¶
If you run a cloud machine, its security is your job. Log in with keys and not passwords, apply updates regularly, open only the network ports you need, keep secrets out of git, and keep backups somewhere else. Test that you can restore them.