Skip to content
Ahmad Ammar

8. Job arrays: a parameter sweep

Say you want to run one program on hundreds of inputs. Submitting hundreds of separate jobs is clumsy. A job array submits them as one, with each task told its number (SchedMD, n.d.).

This chapter builds a working sweep. Download the three files: heat_loss.py, make_cases.py and run_sweep.sh. The "simulation" estimates a building's yearly heating need, but you can swap in your own program.

8.1 The science in one paragraph

Heat flows out through a wall in proportion to its area, how leaky it is (its U-value, in W per m² per kelvin) and the temperature difference. Air leaking out takes heat too: warming 1 m³ of air by 1 K takes about 0.33 Wh. Degree days measure how cold a year is, summing how far each day falls below a base temperature. So heat in kWh per year is:

Q = (U × A + 0.33 × n × V) × DD × 24 / 1000

where A is wall area, n is air changes per hour, V is volume and DD is degree days.

8.2 Step 1: the program

heat_loss.py

#!/usr/bin/env python3
"""Toy simulation for the HPC tutorial: the yearly heating need of one simple building.

    python heat_loss.py case_007.json

It reads one case, does a small calculation, and writes the answer next to the input.
The physics is deliberately basic (see "heat loss" in the tutorial). The numbers are
illustrative: the point is to have something small that you can run thousands of times.
"""
import json   # reads and writes JSON, a plain-text format of names and values
import sys    # gives access to the words typed after the script name (sys.argv)
from pathlib import Path   # handles file paths in a tidy way on any operating system


def yearly_heat(u_wall, wall_area, air_changes, volume, degree_days):
    """Return heat need in kWh per year.

    u_wall       how leaky the wall is, in W per square metre per kelvin
    wall_area    wall area, in square metres
    air_changes  how many times per hour the indoor air is replaced
    volume       indoor volume, in cubic metres
    degree_days  how cold the year is, in kelvin-days
    """
    fabric = u_wall * wall_area                # heat lost through the walls, W per kelvin
    air = 0.33 * air_changes * volume          # heat lost with the air; 0.33 Wh warms 1 m3 of air by 1 K
    return (fabric + air) * degree_days * 24 / 1000   # W/K x K-days x 24 h/day = Wh; /1000 gives kWh


def main():
    path = Path(sys.argv[1])                   # sys.argv[1] is the first word after the script name
    case = json.loads(path.read_text())        # read the file, and turn its text into a Python dictionary
    answer = yearly_heat(**case)               # ** passes the dictionary's entries as named arguments
    # with_suffix swaps .json for .out, so case_007.json gives case_007.out
    path.with_suffix(".out").write_text(f"{answer:.1f}\n")   # :.1f means one decimal place


if __name__ == "__main__":                     # true only when run directly, not when imported
    main()

8.3 Step 2: make the inputs

make_cases.py

#!/usr/bin/env python3
"""Write one input file per combination of settings: a small parameter sweep.

    python make_cases.py cases
"""
import itertools   # itertools.product gives every combination of several lists
import json
import sys
from pathlib import Path

U_WALL = [2.1, 1.5, 0.45]        # three wall types: W per square metre per kelvin
AIR_CHANGES = [1.0, 0.7, 0.5]    # three levels of draughtiness, per hour
DEGREE_DAYS = [1800, 2000, 2200] # three climates, kelvin-days

folder = Path(sys.argv[1])       # the folder name typed after the script name
folder.mkdir(parents=True, exist_ok=True)   # make it, with no error if it exists already

# enumerate numbers the combinations 0, 1, 2 ... so that each file gets a unique name
for number, (u, ach, hdd) in enumerate(itertools.product(U_WALL, AIR_CHANGES, DEGREE_DAYS)):
    case = {"u_wall": u, "wall_area": 90, "air_changes": ach, "volume": 200, "degree_days": hdd}
    # :03d pads the number to three digits, so that files sort in order: 000, 001 ... 026
    (folder / f"case_{number:03d}.json").write_text(json.dumps(case))

print(f"wrote {number + 1} cases to {folder}")

Three settings, each with three values, give 3 × 3 × 3 = 27 combinations.

## cluster, in a project folder holding the three files

## Find your Python first. Look for a module with: module avail 2>&1 | grep -i python
module load <python-module>
python3 make_cases.py cases
ls cases | head                # case_000.json, case_001.json ...
python3 heat_loss.py cases/case_000.json && cat cases/case_000.out    # 11016.0

What you should see

$ python3 make_cases.py cases
wrote 27 cases to cases

$ ls cases | head
case_000.json
case_001.json
case_002.json
case_003.json
case_004.json
case_005.json
case_006.json
case_007.json
case_008.json
case_009.json

$ python3 heat_loss.py cases/case_000.json && cat cases/case_000.out
11016.0

The last line is your test. For the first case, U = 2.1, A = 90, n = 1, V = 200 and DD = 1800, so by hand: (189 + 66) × 1800 × 24 / 1000 = 11016. Then delete that output, with rm cases/case_000.out, so that the job below runs every case.

8.4 Step 3: the job script

Running one input per task works, but very short tasks are wasteful, because starting each one has a cost. So pack several runs into each task.

run_sweep.sh

#!/bin/bash
# Runs a block of cases on one core, as one task of a Slurm job array.
#   sbatch --array=0-5 run_sweep.sh cases 5
# means: 6 tasks (numbered 0 to 5), with 5 cases per task, from the folder "cases".

# Lines starting #SBATCH are read by Slurm. To bash, they are just comments.
#SBATCH --job-name=sweep            # the label shown in the queue
#SBATCH --time=00:10:00             # hard limit, hours:minutes:seconds; the job is stopped when it is reached
#SBATCH --cpus-per-task=1           # one core: this program cannot use more
#SBATCH --mem=1G                    # memory for each task
#SBATCH --output=logs/%x_%A_%a.out  # %x job name, %A array job ID, %a task number; make logs/ first

IN="$1"                             # $1 is the first word after the script name
PER="${2:-5}"                       # $2 is the second word; :-5 means "use 5 if it is missing"

# Fill a bash array named CASES with the sorted file names.
# < <( ... ) feeds the output of a command into mapfile; -t removes the line endings.
mapfile -t CASES < <(ls "$IN"/case_*.json | sort)

# Slurm sets SLURM_ARRAY_TASK_ID for each task. $(( )) does whole-number arithmetic.
# Task 0 starts at case 0, task 1 at case 5, task 2 at case 10, and so on.
start=$(( SLURM_ARRAY_TASK_ID * PER ))

# ${CASES[@]:start:PER} means: PER items of the array, beginning at position start.
for f in "${CASES[@]:start:PER}"; do
  # ${f%.json} removes the ending .json. -f tests whether a file exists.
  # && continue skips to the next case if this one is done, so resubmitting is safe.
  [ -f "${f%.json}.out" ] && continue
  # || runs the echo only if python failed, leaving a line you can search for.
  python3 heat_loss.py "$f" || echo "FAILED $f"
done

echo "task $SLURM_ARRAY_TASK_ID finished its block, starting at case $start on $(hostname)"

Add your module load line just before the mapfile line. Slurm usually passes your session's environment on to the job, but stating it in the script is what makes the job repeatable.

8.5 Step 4: how many tasks?

With N cases and P per task, divide and round up, so that the last few cases are covered. Tasks are numbered from 0. For 27 cases at 5 per task, that is 6 tasks, numbered 0 to 5.

## cluster

## Line endings, if you edited on Windows
sed -i 's/\r$//' run_sweep.sh

## The log folder must exist before submission
mkdir -p logs

## Count the cases, and work out the last task number
N=$(ls cases/case_*.json | wc -l)
PER=5
## In whole-number arithmetic, (N + P - 1) / P rounds up
LAST=$(( (N + PER - 1) / PER - 1 ))
echo "$N cases, tasks 0 to $LAST"

sbatch --array=0-$LAST run_sweep.sh cases $PER

What you should see from the echo line

27 cases, tasks 0 to 5

The sbatch line then prints the number of your job, which differs each time.

A useful extra: --array=0-99%10 runs at most 10 tasks at once, which is polite on a busy system.

8.6 Step 5: did it all work?

## cluster
squeue -u $USER                       # wait until it is empty
ls cases/*.out | wc -l                # expect 27
grep -l FAILED logs/*.out | wc -l     # expect 0
tail -2 logs/sweep_*_0.out            # read one log

If some failed, resubmit the same command. Because each task skips cases that already have an output, only the missing ones run. Being safe to re-run like this makes large runs much less stressful.

8.7 Step 6: collect the results

## cluster
## A loop that prints each case's name and answer into a CSV file.
## ${f%.out} strips the ending, and $(cat "$f") inserts the file's contents
for f in cases/*.out; do echo "$(basename "${f%.out}"),$(cat "$f")"; done > results.csv
head -3 results.csv

What you should see from head

case_000,11016.0
case_001,12240.0
case_002,13464.0
## laptop
scp kelvin:~/project/results.csv "D:\My Work"

8.8 Choosing a packing

Few runs per task (many tasks)

Many runs per task (few tasks)

Finishes

Sooner, if cores are free

Later: each task takes longer

Overhead

Higher: more starts, more load on the scheduler

Lower

After a failure

Less lost

More to redo, unless it is restart-safe

There is no single right answer. Test with your own program, with care: submit each setting separately, repeat it, and keep the copying of files out of your timing. If you are planning a very large run, ask your support team what they prefer.