Workflows/Embarassinly-Parallel

From HPCwiki
Jump to navigation Jump to search

Serial ran one program, once, on one core. This page runs that same program many times over — two hundred samples, a parameter sweep, a hundred simulations that differ only by their random seed — with the runs never needing to speak to each other.

That last part is what makes this the easiest kind of parallelism there is, and by far the most common one in biology. It is also the rung where you most often change nothing whatsoever about your program. The tool stays exactly as it was. What changes is the job script around it, which stops describing one run and starts describing a whole batch of them. Slurm has a purpose-built mechanism for this: the job array.

When this is you

You are looking at an embarrassingly parallel workflow when all three of these are true:

  • You run the same analysis many times over.
  • Each run has its own input — a different sample, a different parameter value, a different seed.
  • No run needs anything another run produces. You could run them in any order, or all at once, and the answers would be identical.

That covers a great deal of everyday work: one pipeline over two hundred sequencing samples, one model fitted per chromosome, one simulation per point on a parameter grid.

The tell that you are not here is a dependency between runs — if run 7 needs a number that run 6 computed, the runs are not independent and an array will not save you. The other tell is subtler: if you have one slow analysis over one dataset, this page cannot help either. An array multiplies runs; it does not split a single run. For that, go to multi-threaded or multi-process.

If you have ever written a for loop that calls sbatch, or submitted the same script by hand with the filename edited each time, this is the page you were looking for.

What it looks like on Anunna

Everything from the serial page still holds: you never run the analysis on the login node, your work lives under $myScratch on Lustre rather than in your home directory, and you hand the job to Slurm with sbatch.

Take the serial script and add one #SBATCH line. The smallest array worth looking at does nothing but show you what the mechanism does: 

#!/bin/bash
#SBATCH --job-name=fruit_salad
#SBATCH --array=0-6            # seven tasks, numbered 0 to 6
#SBATCH --ntasks=1             # each task is still ONE process ...
#SBATCH --cpus-per-task=1      # ... on one core
#SBATCH --mem=100M             # memory PER TASK
#SBATCH --time=00:10:00        # wall-clock limit PER TASK
#SBATCH --output=%x-%A_%a.out  # one log per task

cd "$myScratch/my_analysis"

fruits=(banana apple orange grape berry tomato watermelon)

echo "task ${SLURM_ARRAY_TASK_ID} got: ${fruits[${SLURM_ARRAY_TASK_ID}]}"

Submit it exactly as before, once: 

sbatch run.slurm

Slurm expands that single submission into seven independent jobs. You get seven log files, each naming one fruit. Nothing here is parallel programming — there is no library, no communication, no special compiler. There is a list, and a number that picks one item out of it.

Against the serial script, exactly two things changed:

  • --array=0-6 was added. This is the whole feature. Everything else on this page is a consequence of it.
  • The output name gained %A_%a in place of %j, so each task writes its own log rather than seven tasks fighting over one file.

Everything else — --ntasks, --cpus-per-task, the scratch directory — is the serial script untouched. Each array task is a serial job; there are simply many of them.

Turning the demonstration into real work is a substitution, not a redesign. The list becomes your data, and the echo becomes your program: 

samples=(sampleA sampleB sampleC sampleD)

module load 2024 Python/3.12.3     # replace with your actual module
python myScript.py --input "${samples[${SLURM_ARRAY_TASK_ID}]}.csv" \
                   --output "results/${samples[${SLURM_ARRAY_TASK_ID}]}.out"

myScript.py never learns it is part of an array. It is handed one input and produces one output, exactly as it did serially. That is why this rung is so cheap to climb.

Two numbers behave differently from the multi-threaded page, and both catch people out. --mem and --time are per task: each job gets its own allocation. Threads share memory; array tasks do not, because they are separate jobs on possibly separate nodes. Ask for 4 GB across a 200-task array and you have asked the cluster for 800 GB in total — size the request to one task, then remember to multiply before you are surprised.

The array-specific patterns in the filenames are worth knowing:

Pattern Expands to
%A the array job id — the same for every task
%a the task id — 0, 1, 2 ... the thing that differs
%j the individual job id of that one task
%x the job name

Choosing what each task works on

The array gives every task a number and nothing else. Turning that number into an input is your job. The list above is the clearest way when the list is short; for longer ones there are two more idioms: 

# One line of a file per task — for lists too long to write in the script
SAMPLE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)

# One computed parameter value per task — for sweeps
THRESHOLD=$(awk "BEGIN{print 0.05 * ${SLURM_ARRAY_TASK_ID}}")

Mind the counting, because this is where array jobs go quietly wrong. Bash lists are numbered from 0, so seven fruits are 0-6. sed numbers lines from 1, so two hundred samples are 1-200. Write --array=0-4 against those seven fruits and the job succeeds, reports no error, and simply never processes tomato or watermelon. Nothing fails. A sample is just missing from your final table, and you find out much later, if at all.

Task ids must also be non-negative whole numbers. You cannot pass filenames to --array; you pass positions, and the script looks up the name.

Building the array from the data

Slurm fixes an array's size when the job is submitted, and the script cannot resize its own array from the inside. So when the number of inputs is whatever happens to be in a directory today, compute the range outside the job script and pass it on the command line, where it overrides the #SBATCH directive: 

#!/bin/bash
# launcher.sh — count the data, then submit an array to match

samples=(data/*.csv)
sbatch --array=0-$(( ${#samples[@]} - 1 )) run.slurm

The job script rebuilds the same list with the same glob and indexes into it as before. Note the parentheses in samples=(data/*.csv) — they are what makes this a list. Without them you get a single string, its length is 1 no matter how many files exist, and you submit a one-task array that runs cleanly and does a fraction of the work.

Being a good neighbour

A % at the end of the range caps how many tasks run at once: 

#SBATCH --array=1-200%20       # 200 tasks, at most 20 running concurrently

It is optional, and it is a courtesy worth extending — two hundred tasks all hammering Lustre at the same moment is unpleasant for everyone, including you.

Throttling costs less than you would think, because array tasks queue individually. Each asks for a single core, and single cores come free constantly. A two-hundred-task array typically starts trickling through within minutes, while a job demanding two hundred cores in one block waits for them to line up. This is the same effect described on the overview, working in your favour for once.

The other quiet advantage: when eleven of your two hundred tasks fail, you rerun exactly those eleven. 

sbatch --array=17,42,88-95 run.slurm

What it buys you

Plain arithmetic, and it is the friendliest arithmetic in this section. Two hundred samples at thirty minutes each is one hundred hours — over four days — run one after another. As an array with twenty tasks running at once, it is about five hours. With fifty, about two.

Notice what sets the ceiling. It is not your program and it is not the size of a node: it is how many tasks the cluster is willing to run for you concurrently. Nothing here grows more expensive as you add workers, because the workers never communicate — no message passing, no shared memory, no coordination to pay for. Every other page in this section pays some version of that cost. This one does not, which is why it scales further than any of them.

The π tutorial has no job-array rung — its ladder runs serial, threads, processes, multi-node, GPU — but its discussion makes the general point this page rests on: work that divides into genuinely independent pieces parallelises far better than work that does not.

The honest limitation

It only works if the runs truly do not need each other.

That sounds obvious and is easy to violate by accident. Array tasks do not run in a guaranteed order, and they do not run one at a time — task 5 may finish before task 2 starts. Any assumption about sequence is already broken. So: two tasks writing to the same output file will interleave and corrupt it; two tasks writing scratch files with the same name will clobber each other; a task that reads a result another task is still producing gets whatever happened to be on disk at that instant. None of these fail loudly — they fail by producing plausible, wrong numbers. Give every task its own output path, keyed on $SLURM_ARRAY_TASK_ID, and the whole class of problem disappears.

Real dependencies between runs need a real answer, not an array: MPI if the pieces must exchange data as they compute, or a workflow manager such as Nextflow or Snakemake if the pipeline has stages that must follow one another.

There is one dependency you will almost certainly have, and it is benign: the array leaves you with two hundred output files, and something has to combine them. That collection step is a separate job, usually serial, told to wait for the array to finish: 

ARRAY_ID=$(sbatch --parsable run.slurm)
sbatch --dependency=afterok:${ARRAY_ID} collect.slurm

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=Workflows/Embarassinly-Parallel&oldid=3024"