Workflows/Multi-Process

Multi-threaded work gave one process many arms, and then ran into a wall: threads share memory, and shared memory stops at the edge of the machine. This page is how a single calculation gets past that wall — not by growing one process, but by running many separate processes, each with its own private memory, that send messages to each other as the calculation proceeds. The standard way to do that is MPI, the Message Passing Interface, and it is what lets one job span several nodes of the cluster at once.

This is also the first rung where the program has to have been built for it. On the previous two rungs you changed the job script and the software went along with it. Here the parallelism lives inside the source code, and no #SBATCH line can put it there. So the first question is not how to write the script — it is whether what you are running is an MPI program at all.

When this is you

You are looking at a multi-process workflow when the software was written to run as many cooperating copies of itself:

• Its documentation mentions MPI, or tells you to start it with mpirun, mpiexec, or srun rather than by name.
• It talks about ranks or processes — and about running across nodes — where a threaded tool would talk about threads.
• Installing or loading it pulled in an MPI library: a module with OpenMPI or impi in its name, or a toolchain module such as foss that contains one.

Large simulation codes are the usual inhabitants of this rung — molecular dynamics and computational fluid dynamics especially, with GROMACS and LAMMPS the ones people meet first — along with some assemblers and phylogenetics packages that ship a separate MPI build alongside the ordinary one.

Two tells that you are not here. If the program only offers a --threads-style option, it is multi-threaded and one node is its limit; asking Slurm for tasks it does not know about will not change that. And if what you have is many runs rather than one big one, you want a job array. That distinction is worth being blunt about, because "many processes" describes both: an array also runs many processes at once, but its tasks never speak to each other, while MPI ranks speak constantly. If your runs are genuinely independent, take the array — it is simpler, it starts sooner, and it does not require an MPI program.

What it looks like on Anunna

Everything from the serial job still holds: nothing runs on the login node, work lives under $myScratch on Lustre rather than in your home directory, and the job is a short script handed to sbatch. What changes is which resource lines you raise, and the line that starts the program.

#!/bin/bash
#SBATCH --job-name=my_simulation
#SBATCH --nodes=2                # two machines ...
#SBATCH --ntasks-per-node=64     # ... 64 processes on each: 128 ranks in total
#SBATCH --cpus-per-task=1        # each rank single-threaded
#SBATCH --mem-per-cpu=2G         # memory PER CORE, not per node -- see below
#SBATCH --time=01:00:00          # wall-clock limit; the job is killed past this
#SBATCH --output=%x-%j.out       # stdout + stderr, named by job name and id

cd "$myScratch/my_simulation"

module load 2024 foss/2024a      # GCC + OpenMPI; match how the program was built

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc

Set against the serial script, the interesting part is which number went up:

• --ntasks is what grew — expressed here as --nodes × --ntasks-per-node, which comes to 128 — while --cpus-per-task stayed at 1. That is the exact mirror image of the multi-threaded page, which raised --cpus-per-task and left --ntasks at 1. Those two lines are the whole difference between the two rungs: one process with many cores, or many processes with one core each.
• --nodes appears for the first time in this section. It can, because ranks do not share memory and therefore do not have to share a machine. You may also simply write --ntasks=128 and let Slurm place the ranks wherever it likes; spelling out nodes and tasks-per-node instead gives you predictable placement, which makes timings comparable between runs.
• The launch line hands Slurm's number to the program. Slurm exports the task count as SLURM_NTASKS, and mpirun -np ${SLURM_NTASKS} starts exactly that many ranks. This is the same habit as OMP_NUM_THREADS one rung down: never hard-code the number in two places, or the request and the run will eventually disagree.

Which MPI module

The rule that matters: load the same MPI the program was built against. Mixing them — building with one MPI and running with another — produces errors that look like almost anything except the mismatch causing them.

If the software came from the module system, loading its module is the reliable route, because that pulls in the matching MPI as a dependency and you never choose by hand. If you are running something you built yourself, load the toolchain you built it with — and if you are the one doing the building, that is what the MPI module's compiler wrappers (mpicc, mpicxx, mpifort) are for: they call your ordinary compiler with the MPI headers and libraries already in place. The π tutorial's MPI rung uses:

module purge
module load 2024 foss/2024a      # GCC 13.3 with OpenMPI 5.0.3

Modules on Anunna sit inside year buckets, and a bucket has to be loaded before its contents are visible — so module avail OpenMPI will show you nothing useful until you have loaded a 2024 or 2025 first. Newer MPI versions live in the newer buckets. See Environment Modules for how the buckets work. Note also that module purge removes the slurm module along with everything else; reload it if your script needs Slurm commands after purging.

mpirun or srun

Both will start an MPI program, and you will meet both in documentation. The material in this section uses mpirun consistently, for a practical reason: it is OpenMPI's own launcher, so OpenMPI's placement options behave as its documentation says they do, whereas launching through srun adds a second layer that can quietly override them. Unless you have a reason to prefer srun, use mpirun -np ${SLURM_NTASKS} and stay with it.

Memory is counted per rank now

On the serial page, --mem=4G meant four gigabytes for the job's one process. It does not mean anything like that here, and this is the mistake most likely to kill an otherwise correct MPI job.

--mem is a request per node, and it is shared out among every rank running on that node. Ask for --mem=1G and put 128 ranks on the node, and each rank gets roughly eight megabytes — less than an MPI process needs merely to start up, before your data is anywhere in the picture. The job dies of an out-of-memory kill early in the run, and nothing in the error message points back at the flag responsible. This is not a hypothetical: it is precisely what happened while the π tutorial's MPI rungs were being built, and switching to --mem-per-cpu was the fix.

So on this rung, think per rank and write --mem-per-cpu. Multiply in your head to check the total is sane — --mem-per-cpu=2G with 64 ranks on a node is 128 GB from that node, which is comfortable on Anunna's larger nodes and impossible on its smaller ones.

What it buys you

The π tutorial runs the same ten-billion-dart estimate as an MPI program, on one node and across four, alongside the serial baseline:

variant   nodes cores    wall_s
serial        1     1    26.008
mpi           1   128     0.555
mpi           4    64     1.090

A hundred and twenty-eight ranks turn 26 seconds into about half a second. The more instructive pair is the second and third rows, which look at first glance like a penalty for leaving the node — until you notice the four-node run used 64 cores against the single node's 128. Compared like with like, the tutorial's 64-core runs took about 1.11 s on one node and about 1.09 s spread across four: the same, within noise. For this workload, crossing the network cost nothing measurable.

That result is genuinely good news and should still be read carefully, because this problem barely communicates — the ranks work alone and exchange one number each at the very end. A real simulation exchanges data at every step, and that cost grows as ranks are added, which is why real codes have a scaling sweet spot rather than a straight line and why the only trustworthy answer comes from measuring your program on your data. The tutorial has a wrinkle in the same spirit: its single-rank MPI run takes about 70 seconds where the single-thread OpenMP run takes 26, a property of how that particular program starts up rather than a rule about MPI. Small scale is where MPI's overheads are most visible and least repaid.

The honest limitation

The program must have been written for MPI, and you cannot bolt it on. No #SBATCH line, no module, no wrapper script adds MPI to software that does not already have it. Run a non-MPI program under mpirun -np 128 and you do not get a parallel run; you get 128 identical copies of the same serial calculation, each ignorant of the others, quite possibly writing over each other's output files. It looks busy and it is worse than useless.

If the documentation never mentions MPI or ranks, that is your answer, and it is a perfectly ordinary one — most software is not MPI software. Go back a rung to multi-threaded, or sideways to a job array if you have many runs rather than one.

And when you do have an MPI program, more ranks is not automatically faster. Communication is real work, the queue makes you wait longer for 256 cores than for 64, and past some rank count your run stops improving and starts getting worse. Find where that happens on a short test case before committing a long job to it.

If your MPI program can also use threads inside each rank — many of the big codes can — the combination is usually faster than either alone, and it is its own page: Hybrid Workflows.

See also

• Parallel Computing — the overview, and the map to the other workflow types.
• Multi-Threaded Workflows — the previous rung, and the one-node ceiling this page breaks.
• Hybrid Workflows — MPI between nodes, threads within them.
• Embarrassingly Parallel — for many independent runs, which need no MPI at all.
• Parallelism: Estimating π — its MPI and multi-node rungs are the worked example for this page.
• Environment Modules — year buckets, and how to find the MPI you need.
• Compute Hardware Overview — how many cores and how much memory a node actually has.