OpenMPI

OpenMPI is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node.

This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport, which on Anunna needs saying out loud. If you are still deciding whether MPI is the right shape for your work at all, start at Multi-Process Workflows — that page covers when to use MPI and when a job array or threads would serve you better. For a worked example you can run yourself, see Parallelism: Estimating π.

One thing to be clear about before anything else: MPI has to be built into the program. You cannot add it from a job script. If your software's documentation never mentions MPI or ranks, this page is not the one you need.

What is available on Anunna

Software here is built with EasyBuild and grouped into buckets. A bucket has to be loaded before you can load anything from it — and before module avail will list its contents. Each bucket corresponds to an EasyBuild toolchain generation:

Bucket	Toolchain	OpenMPI	Compiler
2023	foss/2023a, gompi/2023a	OpenMPI/4.1.5	GCC 12.3.0
2024	foss/2024a, gompi/2024a	OpenMPI/5.0.3	GCC 13.3.0
2025	foss/2025a, gompi/2025a	OpenMPI/5.0.7	GCC 14.2.0

Three ways in, and they differ only in how much they bring with them:

• foss — the full toolchain: GCC, OpenMPI, OpenBLAS, FFTW and ScaLAPACK. Load this if your program wants linear algebra or FFTs as well as MPI.
• gompi — GCC and OpenMPI, nothing else. The lean choice when MPI is all you need.
• OpenMPI/<version> — the library on its own, with its compiler underneath.

You can list them yourself with module key, which searches module descriptions as well as names. It reads Lmod's cache rather than your current environment, so — unlike module avail — it needs no bucket loaded and reports matches from every bucket at once:

module key OpenMPI        # no bucket needed; searches across all of them

That is the quickest way to see which OpenMPI versions exist and which toolchains carry them. Once you have picked one, then load its bucket and the module.

Loading it

Two steps, always in this order: the bucket, then the module. Nothing from a bucket can be loaded until the bucket itself is.

To get the full toolchain from the 2024 bucket:

module load 2024
module load 2024 OpenMPI/5.0.3    # OpenMPI alone

The short version number is enough — you do not need to type the compiler suffix. OpenMPI/5.0.3 resolves to OpenMPI/5.0.3-GCC-13.3.0, which is the full name module list will show you afterwards. If you want to see everything a bucket offers, module avail OpenMPI lists it once that bucket is loaded.

Alternatively, swap the second line for modules that bundle OpenMPI with other additional modules:

module load 2024 gompi/2024a      # GCC + OpenMPI, no maths libraries
module load foss/2024a            # GCC + OpenMPI + maths libraries

Check what you actually got. Two commands, both worth the habit

module list        # everything now loaded, including pulled-in dependencies
mpirun --version   # confirms which OpenMPI is on your PATH

Which one to load

The rule that saves the most time: load the same MPI the program was built against. Building with one MPI and running under another produces failures that look like almost anything except the mismatch causing them — hangs, garbled output, crashes inside library calls.

In practice that means:

• Running software from the module system? Load its module and let it pull in the matching MPI as a dependency. You never have to choose, and you cannot get it wrong.
• Running something you built yourself? Load the same toolchain you built it with.
• Building it yourself? Use the compiler wrappers the MPI module provides — mpicc, mpicxx, mpifort — which call your ordinary compiler with the MPI headers and libraries already in place.

If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.

Recommended Environment

export OMPI_MCA_pml=cm
export OMPI_MCA_mtl=ofi
export FI_PROVIDER=psm2
export OMPI_MCA_btl=self,sm #substitute sm with vader for v4

Running on a single node

Remember that nothing should run on the login nodes — they are shared, and are for editing, submitting and light housekeeping only. Describe the job in a script, keep the data on Lustre under $myScratch, and hand it to sbatch.

#!/bin/bash
#SBATCH --job-name=mpi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64             # 64 ranks, all on one machine
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G        # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

export OMPI_MCA_pml=cm
export OMPI_MCA_mtl=ofi
export FI_PROVIDER=psm2
export OMPI_MCA_btl=self,sm

mpirun -np ${SLURM_NTASKS} ./my_mpi_program

Two habits in there are worth keeping:

• -np ${SLURM_NTASKS}, never a hard-coded number. Slurm exports the task count it granted; passing that through means the request and the run cannot drift apart when you change one of them.
• --mem-per-cpu, not --mem. --mem is a per-node request shared out among all ranks on that node — at 64 or 128 ranks each one ends up with a few megabytes, less than an MPI process needs simply to start, and the job dies of an out-of-memory kill that points nowhere near the flag responsible.

On one node OpenMPI talks to itself through shared memory. There is nothing to configure and no network involved, which is why this case is simple and the next one is not.

Running across several nodes

The moment a job spans two nodes, the ranks have to talk over the network — and on Anunna that path needs to be chosen explicitly.

Anunna's fabric is Omni-Path (OPA100), and inter-node MPI rides it through libfabric. libfabric offers more than one provider for this hardware, and on our cluster they are not equally healthy:

Provider	Status on Anunna	Use it?
psm2	Validated at line rate (~97 Gbps), on both Intel and AMD nodes, under all three toolchains	Yes — this is the one
opx	Not currently operational	Not yet

Because TCP can be picked up automatically, an inter-node MPI job should explicitly ask for psm2 rather than trusting the default:

mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \
       -x FI_PROVIDER=psm2 \
       -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc

A complete multi-node job script:

#!/bin/bash
#SBATCH --job-name=mpi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64    # 256 ranks in total
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

mpirun --mca pml cm --mca mtl ofi \
       --mca mtl_ofi_provider_include psm2 \
       --mca btl self,sm \
       -x FI_PROVIDER=psm2 \
       -np ${SLURM_NTASKS} ./my_mpi_program

Note some of the settings can be done by OpenMPI's own tags.

Exporting variables to the other nodes: the -x flag

-x hands a named environment variable to every rank, including those in other nodes :

mpirun -x FI_PROVIDER=psm2 -x MY_DATA_DIR -np ${SLURM_NTASKS} ./prog

-x VAR=value sets it; -x VAR passes through whatever value it already has. Use one -x per variable.

With OpenMPI 5 inside a Slurm job, you usually do not need it. Though, do not read that as universal, it depends on the version. It was verified for OpenMPI 5.0.7, in the 2025 bucket. The 2023 bucket carries OpenMPI 4.1.5, which may not propagate your environment automatically; historically variables did have to be passed by hand. And in a hostfile launch over ssh, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

So use -x if you are on the 2023 bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

So treat -x as the explicit form rather than a workaround. It costs nothing, it states in the launch line what the run depends on, and it keeps working if the launch path changes underneath you. And if a job behaves differently across several nodes than it does on one, a variable that failed to travel is still the first thing worth suspecting.

A start-up gotcha

Do not use sbatch --wrap for MPI jobs. The wrapped command runs under dash, where Lmod's module function is not initialised, so module loads fail quietly and the job proceeds without the software it asked for. Write a real script starting #!/bin/bash -l instead.

Advanced: pinning and rank placement

By default OpenMPI decides where each rank sits and which cores it may use. For most jobs that is fine. It stops being fine when a job is memory-bandwidth-bound, when ranks each run threads of their own, or when timings vary between runs for no visible reason — all of which come down to placement.

Two ideas, easy to mix up:

• Mapping — which rank goes where. --map-by <unit>, where the unit can be core, l3cache, numa, package (socket), or node.
• Binding — how tightly a rank is then pinned. --bind-to core is the usual choice; --bind-to none lets a rank roam.

Before tuning anything, look at what you are actually getting:

mpirun --report-bindings -np ${SLURM_NTASKS} ./my_mpi_program

That prints one line per rank showing the cores it is bound to. It is the only way to know rather than assume, and defaults differ between OpenMPI versions.

Why multiples of eight suit our AMD nodes

Most of Anunna's compute is AMD — Zen 3 (EPYC Milan) and Zen 5 (EPYC Turin). Both build a socket out of chiplets of 8 cores, and each chiplet has its own slice of L3 cache. Cores inside a chiplet share that cache and reach memory together; cores in different chiplets do not.

The practical consequence: ask for ranks in multiples of 8, so that chiplets are filled evenly and no chiplet is left holding a single stray rank while its cache and memory channels go mostly unused. 64, 128 and 256 are comfortable numbers here; 50 or 100 are not, and will spread unevenly no matter how you map them.

Mapping by cache makes the intent explicit — one rank per chiplet, each free to use that chiplet's cores:

mpirun --map-by l3cache --bind-to core -np ${SLURM_NTASKS} ./my_mpi_program

A related point for memory-bound work: spreading ranks across chiplets and NUMA domains uses many memory controllers at once, while packing them close leaves most idle. On our AMD nodes that difference is large, not marginal.

Ranks with threads inside them

If each rank also runs OpenMP threads, mapping and binding stop being optional. Give each rank a block of cores with PE=, and bind inside it:

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
mpirun --map-by ppr:8:node:PE=${SLURM_CPUS_PER_TASK} --bind-to core \
       -x OMP_NUM_THREADS \
       -np ${SLURM_NTASKS} ./my_hybrid_program

That reads as: place 8 ranks per node, give each one SLURM_CPUS_PER_TASK cores, and bind them there. Without the PE= part every thread of a rank can end up on one single core — the job runs, produces correct results, and is many times slower than it should be. It is a silent failure, so check it with --report-bindings.

Note also that MPI binding and OpenMP affinity are separate knobs. --map-by and --bind-to decide which cores a rank owns; OMP_PROC_BIND and OMP_PLACES decide how that rank's threads arrange themselves within those cores. Setting the OpenMP variables cannot rescue a rank that was only given one core to begin with — the threads have nowhere to spread to. Get the MPI side right first.

See also

• Multi-Process Workflows — when MPI is the right choice, and what the job script looks like.
• Intel MPI — the other MPI here, its I_MPI_* pinning dialect, and MKL's behaviour on AMD nodes.
• Multi-threaded Workflows — threads on one node, for programs that are not MPI.
• Parallelism: Estimating π — the same calculation run serially, threaded, with MPI, and across nodes.
• Environment Modules — buckets, and how to find the module you need.
• Compute Hardware Overview — the nodes, their core counts and their memory.
• Batch Jobs — writing and submitting job scripts.
• Scheduler Overview (Slurm) — how the scheduler allocates what you ask for.