Honfi001 at 14:46, 5 August 2026

2026-08-05T14:46:12Z

← Older revision		Revision as of 14:46, 5 August 2026
Line 44:		Line 44:
	module list # everything loaded, dependencies included		module list # everything loaded, dependencies included
	mpirun -version # which Intel MPI is on your PATH		mpirun -version # which Intel MPI is on your PATH
	~~</syntaxhighlight>~~

	If your software has its own module, load '''that''' instead and let it pull the matching MPI in as a dependency. A package built with this toolchain has a name ending in <code>-intel-2024a</code> or similar, which tells you what you are getting:

	~~<syntaxhighlight lang="bash">~~
	~~module load 2024~~
	~~module load <YourSoftware>/<version> # Intel MPI arrives as a dependency~~
	</syntaxhighlight>		</syntaxhighlight>

Line 138:		Line 131:
	'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.		'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.

	== Intel MPI, MKL, and ~~our~~ AMD nodes ==		== Intel MPI, MKL, and AMD nodes ==

	Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.		Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.
Line 151:		Line 144:
	</syntaxhighlight>		</syntaxhighlight>

	Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound ~~Level-3~~ work''' ~~such as <code>DGEMM</code>~~. It does nothing when MKL is only handling bandwidth-bound ~~Level-1/2 vector kernels~~, and nothing at all if your library does its own arithmetic — PETSc~~, for instance, performs its sparse matrix–vector products itself, so MKL never sees the dominant cost of a Krylov solve~~.		Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound work'''. It does nothing when MKL is only handling bandwidth-bound work, and nothing at all if your library does its own arithmetic e.g. PETSc.

	<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.		<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.

Honfi001: Created page with "'''Intel MPI''' is the other MPI implementation available on Anunna, alongside OpenMPI. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL). Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on Multi-Process Workflows. This page is the Intel-sp..."

2026-08-05T14:40:48Z

Created page with "'''Intel MPI''' is the other MPI implementation available on Anunna, alongside OpenMPI. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL). Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on Multi-Process Workflows. This page is the Intel-sp..."

New page

'''Intel MPI''' is the other MPI implementation available on Anunna, alongside [[OpenMPI]]. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL).

Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on [[Workflows/Multi-Process|Multi-Process Workflows]]. This page is the Intel-specific practice: which modules exist, how to launch, how the pinning controls differ from OpenMPI's, and one thing that catches people out on our AMD nodes.

'''Which should you use?''' Whichever your program was built with. That is not a dodge — mixing MPI implementations between build and run does not work, so in practice the software chooses for you. Where you genuinely have a choice, Intel MPI and MKL are the natural pairing for code that leans on Intel's maths libraries, and [[OpenMPI]] is the default for everything else here.

== What is available on Anunna ==

Software is built with [https://easybuild.io EasyBuild] and grouped into [[Environment Modules|buckets]]; a bucket has to be loaded before you can load anything from it. The Intel toolchain follows the same yearly generations as <code>foss</code>:

{| class="wikitable"
! Bucket !! Toolchain !! Contains
|-
| <code>2023</code> || <code>intel/2023a</code> || Intel compilers, Intel MPI, MKL
|-
| <code>2024</code> || <code>intel/2024a</code> || Intel compilers, Intel MPI, MKL
|-
| <code>2025</code> || <code>intel/2025a</code> || Intel compilers, Intel MPI, MKL
|}

<code>intel/<year>a</code> is the counterpart of <code>foss/<year>a</code>: one module that brings the whole stack — compilers, MPI, MKL and FFTW. There are leaner ways in, mirroring <code>gompi</code> on the <code>foss</code> side:

* '''<code>iimpi/<year>a</code>''' — Intel compilers and Intel MPI, without MKL. The lean choice when you only need MPI.
* '''<code>impi/<version></code>''' — the MPI library alone. In the 2025 bucket this is <code>impi/2021.15.0-intel-compilers-2025.1.1</code>; note that Intel MPI's own version numbering (2021.x) does not track the toolchain year.

To see what a bucket actually offers, ask the module system — <code>module key</code> reads Lmod's cache, so it needs no bucket loaded and searches every bucket at once:

<syntaxhighlight lang="bash">
module key intel # the toolchains
module key impi # the MPI library on its own
</syntaxhighlight>

=== Loading it ===

Bucket first, then the module — the same two steps as any other software here:

<syntaxhighlight lang="bash">
module load 2024 intel/2024a
</syntaxhighlight>

Then confirm what you got, which is worth the habit:

<syntaxhighlight lang="bash">
module list # everything loaded, dependencies included
mpirun -version # which Intel MPI is on your PATH
</syntaxhighlight>

If your software has its own module, load '''that''' instead and let it pull the matching MPI in as a dependency. A package built with this toolchain has a name ending in <code>-intel-2024a</code> or similar, which tells you what you are getting:

<syntaxhighlight lang="bash">
module load 2024
module load <YourSoftware>/<version> # Intel MPI arrives as a dependency
</syntaxhighlight>

Keep a job inside one bucket. Do not load <code>intel</code> and <code>foss</code> in the same job — they provide competing compilers, MPI libraries and BLAS, and the result is unpredictable rather than merely slow. If you want a clean starting point, <code>module reset</code> returns you to the system defaults; the <code>slurm</code> module is ''sticky'' and survives it either way.

== Running on a single node ==

Nothing runs on the login nodes. Put the work in a script, keep data on Lustre under <code>$myScratch</code>, and submit with <code>sbatch</code>.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=impi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

Two things carried over from [[OpenMPI]], for the same reasons: pass <code>${SLURM_NTASKS}</code> rather than hard-coding the rank count, and use <code>--mem-per-cpu</code> rather than <code>--mem</code>, which is a per-''node'' request divided among all the ranks on that node.

Note the <code>#!/bin/bash -l</code>. A login shell is what initialises Lmod, so without it <code>module load</code> can fail quietly. For the same reason, never launch an MPI job with <code>sbatch --wrap</code> — that runs under <code>dash</code>, where <code>module</code> does not exist.

On one node the ranks talk through shared memory and there is nothing to configure.

== Running across several nodes ==

Anunna's fabric is '''Omni-Path (OPA100)''', and both MPI implementations reach it through '''libfabric'''. The provider that works reliably here is '''<code>psm2</code>'''; the <code>opx</code> provider is broken for inter-node traffic on this hardware. So an inter-node job should name the provider rather than trust auto-selection:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=impi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -genv I_MPI_OFI_PROVIDER psm2 -genv FI_PROVIDER psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

To see what was actually selected — provider, pinning, rank layout — turn up Intel MPI's own diagnostics:

<syntaxhighlight lang="bash">
export I_MPI_DEBUG=4
</syntaxhighlight>

That prints the chosen fabric and the rank-to-core map at start-up, and is the fastest way to confirm a setting took effect instead of assuming it did.

=== Getting variables to the other nodes ===

Ranks on remote nodes are started fresh, so any variable your program or the fabric depends on has to reach them. Intel MPI's controls are:

{| class="wikitable"
! Option !! Effect
|-
| <code>-genv <VAR> <value></code> || Set one variable for '''all''' ranks. The explicit, always-safe form.
|-
| <code>-genvall</code> || Pass the whole launching environment to all ranks. This is the default.
|-
| <code>-genvlist <a,b,c></code> || Pass only the named variables.
|-
| <code>-genvnone</code> || Pass nothing.
|}

In practice '''a plain <code>export</code> in your job script does reach every rank''' — tested on Anunna across two nodes.

It arrives whatever you do to stop it: restricting Intel MPI's own propagation with <code>-genvlist PATH</code>, which should have excluded the test variable, delivered it to both nodes anyway. [[OpenMPI]] 5.0.7 behaved identically — with <code>-x</code>, without it, and even with Slurm's own environment forwarding suppressed.

The practical reading is that '''inside a Slurm allocation both implementations get your environment to the ranks''', by way of the process-management layer (PMIx and the launcher's job description) rather than the batch environment. On the OpenMPI side that was pinned down: removing the variable from <code>mpirun</code>'s own environment is the only thing that stopped it arriving — see [[OpenMPI#Exporting variables to the other nodes: the -x flag|the OpenMPI page]] for the full result.

None of which is worth depending on in a job script. It held for the current toolchains and not necessarily for older ones, so say what your run needs explicitly with <code>-genv</code> and the question stops mattering.

'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.

== Intel MPI, MKL, and our AMD nodes ==

Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.

MKL checks the '''CPU vendor''' at run time. On a non-Intel CPU, kernels without specific Zen coverage can fall back to an '''SSE''' code path instead of using '''AVX2''', leaving much of the vector width unused. This is still the behaviour in MKL 2025.x — it has not been quietly fixed. The old <code>MKL_DEBUG_CPU_TYPE</code> workaround is gone, removed back in MKL 2020 Update 1, so anything you read recommending it is out of date.

The current mitigation is a small shim, '''<code>libfakeintel.so</code>''', <code>LD_PRELOAD</code>ed ahead of MKL. It overrides the vendor test (<code>mkl_serv_intel_cpu_true</code>) so that it returns true, and MKL then dispatches its AVX2 kernels:

<syntaxhighlight lang="bash">
export LD_PRELOAD=/path/to/libfakeintel.so
mpirun -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound Level-3 work''' such as <code>DGEMM</code>. It does nothing when MKL is only handling bandwidth-bound Level-1/2 vector kernels, and nothing at all if your library does its own arithmetic — PETSc, for instance, performs its sparse matrix–vector products itself, so MKL never sees the dominant cost of a Krylov solve.

<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.

If MKL turns out to suit your workload poorly on the AMD nodes, the <code>foss</code> toolchain's OpenBLAS-based BLAS is worth benchmarking against.

== Advanced: pinning and rank placement ==

Intel MPI and OpenMPI express placement differently: OpenMPI takes command-line flags, Intel MPI reads <code>I_MPI_*</code> environment variables. The intents map across cleanly, so if you know one dialect this table gives you the other:

{| class="wikitable"
! Intent !! Intel MPI !! OpenMPI
|-
| One rank per physical core, no migration || <code>I_MPI_PIN=1</code>, <code>I_MPI_PIN_DOMAIN=core</code> || <code>--bind-to core</code>
|-
| Fill the node, neighbours adjacent || <code>I_MPI_PIN_ORDER=compact</code> || <code>--map-by core</code>
|-
| Under-subscribed, spread across all NUMA domains || <code>I_MPI_PIN_ORDER=scatter</code> || <code>--map-by numa --bind-to core</code>
|-
| One rank per chiplet / L3 slice || <code>I_MPI_PIN_DOMAIN=cache3</code> || <code>--map-by l3cache --bind-to core</code>
|-
| Print the map actually used || <code>I_MPI_DEBUG=4</code> || <code>--report-bindings</code>
|}

Three rules that matter more than the individual settings:

* '''Verify, do not assume.''' Print the map with <code>I_MPI_DEBUG=4</code> before and after any change. Placement bugs do not announce themselves; the job runs and is simply slower than it should be.
* '''One source of truth.''' Pin with the MPI launcher '''or''' with Slurm (<code>--cpu-bind</code>, <code>--distribution</code>), never both. Two pinners fighting each other produce a nonsense map.
* '''Spread suits memory-bound work.''' AMD sockets are built from 8-core chiplets, each with its own cache slice and share of the memory channels. A bandwidth-bound job saturates memory well before every core is busy, so spreading a reduced number of ranks across all NUMA domains keeps every memory channel active, while packing them together leaves most idle. Asking for ranks in multiples of 8 keeps chiplets evenly filled.

For the reasoning behind that last point, and the equivalent OpenMPI syntax, see [[OpenMPI#Advanced: pinning and rank placement|the OpenMPI page]].

== See also ==

* [[OpenMPI]] — the other MPI here, and the default for most software.
* [[Workflows/Multi-Process|Multi-Process Workflows]] — whether MPI is the right shape for your work.
* [[Environment Modules]] — buckets, and finding the module you need.
* [[Compute Hardware Overview]] — what the nodes actually are.
* [[Batch Jobs]] — writing and submitting job scripts.
* [[Scheduler Overview (Slurm)]] — how the scheduler allocates what you ask for.

IntelMPI - Revision history

Honfi001 at 14:46, 5 August 2026