IntelMPI

From HPCwiki
Revision as of 14:46, 5 August 2026 by Honfi001 (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Intel MPI is the other MPI implementation available on Anunna, alongside OpenMPI. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL).

Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on Multi-Process Workflows. This page is the Intel-specific practice: which modules exist, how to launch, how the pinning controls differ from OpenMPI's, and one thing that catches people out on our AMD nodes.

Which should you use? Whichever your program was built with. That is not a dodge — mixing MPI implementations between build and run does not work, so in practice the software chooses for you. Where you genuinely have a choice, Intel MPI and MKL are the natural pairing for code that leans on Intel's maths libraries, and OpenMPI is the default for everything else here.

What is available on Anunna

Software is built with EasyBuild and grouped into buckets; a bucket has to be loaded before you can load anything from it. The Intel toolchain follows the same yearly generations as foss:

Bucket Toolchain Contains
2023 intel/2023a Intel compilers, Intel MPI, MKL
2024 intel/2024a Intel compilers, Intel MPI, MKL
2025 intel/2025a Intel compilers, Intel MPI, MKL

intel/<year>a is the counterpart of foss/<year>a: one module that brings the whole stack — compilers, MPI, MKL and FFTW. There are leaner ways in, mirroring gompi on the foss side:

  • iimpi/<year>a — Intel compilers and Intel MPI, without MKL. The lean choice when you only need MPI.
  • impi/<version> — the MPI library alone. In the 2025 bucket this is impi/2021.15.0-intel-compilers-2025.1.1; note that Intel MPI's own version numbering (2021.x) does not track the toolchain year.

To see what a bucket actually offers, ask the module system — module key reads Lmod's cache, so it needs no bucket loaded and searches every bucket at once: 

module key intel      # the toolchains
module key impi       # the MPI library on its own

Loading it

Bucket first, then the module — the same two steps as any other software here: 

module load 2024 intel/2024a

Then confirm what you got, which is worth the habit: 

module list           # everything loaded, dependencies included
mpirun -version       # which Intel MPI is on your PATH

Keep a job inside one bucket. Do not load intel and foss in the same job — they provide competing compilers, MPI libraries and BLAS, and the result is unpredictable rather than merely slow. If you want a clean starting point, module reset returns you to the system defaults; the slurm module is sticky and survives it either way.

Running on a single node

Nothing runs on the login nodes. Put the work in a script, keep data on Lustre under $myScratch, and submit with sbatch. 

#!/bin/bash -l
#SBATCH --job-name=impi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G        # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc

Two things carried over from OpenMPI, for the same reasons: pass ${SLURM_NTASKS} rather than hard-coding the rank count, and use --mem-per-cpu rather than --mem, which is a per-node request divided among all the ranks on that node.

Note the #!/bin/bash -l. A login shell is what initialises Lmod, so without it module load can fail quietly. For the same reason, never launch an MPI job with sbatch --wrap — that runs under dash, where module does not exist.

On one node the ranks talk through shared memory and there is nothing to configure.

Running across several nodes

Anunna's fabric is Omni-Path (OPA100), and both MPI implementations reach it through libfabric. The provider that works reliably here is psm2; the opx provider is broken for inter-node traffic on this hardware. So an inter-node job should name the provider rather than trust auto-selection: 

#!/bin/bash -l
#SBATCH --job-name=impi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -genv I_MPI_OFI_PROVIDER psm2 -genv FI_PROVIDER psm2 \
       -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc

To see what was actually selected — provider, pinning, rank layout — turn up Intel MPI's own diagnostics: 

export I_MPI_DEBUG=4

That prints the chosen fabric and the rank-to-core map at start-up, and is the fastest way to confirm a setting took effect instead of assuming it did.

Getting variables to the other nodes

Ranks on remote nodes are started fresh, so any variable your program or the fabric depends on has to reach them. Intel MPI's controls are:

Option Effect
-genv <value> Set one variable for all ranks. The explicit, always-safe form.
-genvall Pass the whole launching environment to all ranks. This is the default.
-genvlist <a,b,c> Pass only the named variables.
-genvnone Pass nothing.

In practice a plain export in your job script does reach every rank — tested on Anunna across two nodes.

It arrives whatever you do to stop it: restricting Intel MPI's own propagation with -genvlist PATH, which should have excluded the test variable, delivered it to both nodes anyway. OpenMPI 5.0.7 behaved identically — with -x, without it, and even with Slurm's own environment forwarding suppressed.

The practical reading is that inside a Slurm allocation both implementations get your environment to the ranks, by way of the process-management layer (PMIx and the launcher's job description) rather than the batch environment. On the OpenMPI side that was pinned down: removing the variable from mpirun's own environment is the only thing that stopped it arriving — see the OpenMPI page for the full result.

None of which is worth depending on in a job script. It held for the current toolchains and not necessarily for older ones, so say what your run needs explicitly with -genv and the question stops mattering.

-genv is still the habit worth keeping for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.

Intel MPI, MKL, and AMD nodes

Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: the issue is in MKL, not in Intel MPI. But MKL arrives with the same intel/<year>a module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.

MKL checks the CPU vendor at run time. On a non-Intel CPU, kernels without specific Zen coverage can fall back to an SSE code path instead of using AVX2, leaving much of the vector width unused. This is still the behaviour in MKL 2025.x — it has not been quietly fixed. The old MKL_DEBUG_CPU_TYPE workaround is gone, removed back in MKL 2020 Update 1, so anything you read recommending it is out of date.

The current mitigation is a small shim, libfakeintel.so, LD_PRELOADed ahead of MKL. It overrides the vendor test (mkl_serv_intel_cpu_true) so that it returns true, and MKL then dispatches its AVX2 kernels: 

export LD_PRELOAD=/path/to/libfakeintel.so
mpirun -np ${SLURM_NTASKS} ./my_mpi_program

Before assuming this is your problem, two things are worth checking, because often it is not. The shim only widens arithmetic, so it pays off when MKL is doing compute-bound work. It does nothing when MKL is only handling bandwidth-bound work, and nothing at all if your library does its own arithmetic e.g. PETSc.

MKL_VERBOSE=1 is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.

If MKL turns out to suit your workload poorly on the AMD nodes, the foss toolchain's OpenBLAS-based BLAS is worth benchmarking against.

Advanced: pinning and rank placement

Intel MPI and OpenMPI express placement differently: OpenMPI takes command-line flags, Intel MPI reads I_MPI_* environment variables. The intents map across cleanly, so if you know one dialect this table gives you the other:

Intent Intel MPI OpenMPI
One rank per physical core, no migration I_MPI_PIN=1, I_MPI_PIN_DOMAIN=core --bind-to core
Fill the node, neighbours adjacent I_MPI_PIN_ORDER=compact --map-by core
Under-subscribed, spread across all NUMA domains I_MPI_PIN_ORDER=scatter --map-by numa --bind-to core
One rank per chiplet / L3 slice I_MPI_PIN_DOMAIN=cache3 --map-by l3cache --bind-to core
Print the map actually used I_MPI_DEBUG=4 --report-bindings

Three rules that matter more than the individual settings:

  • Verify, do not assume. Print the map with I_MPI_DEBUG=4 before and after any change. Placement bugs do not announce themselves; the job runs and is simply slower than it should be.
  • One source of truth. Pin with the MPI launcher or with Slurm (--cpu-bind, --distribution), never both. Two pinners fighting each other produce a nonsense map.
  • Spread suits memory-bound work. AMD sockets are built from 8-core chiplets, each with its own cache slice and share of the memory channels. A bandwidth-bound job saturates memory well before every core is busy, so spreading a reduced number of ranks across all NUMA domains keeps every memory channel active, while packing them together leaves most idle. Asking for ranks in multiples of 8 keeps chiplets evenly filled.

For the reasoning behind that last point, and the equivalent OpenMPI syntax, see the OpenMPI page.

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=IntelMPI&oldid=3035"