OpenMPI - Revision history

Honfi001 at 14:55, 6 August 2026

2026-08-06T14:55:29Z

← Older revision		Revision as of 14:55, 6 August 2026
Line 92:		Line 92:

	module load 2024 foss/2024a		module load 2024 foss/2024a

			export OMPI_MCA_pml=cm
			export OMPI_MCA_mtl=ofi
			export FI_PROVIDER=psm2
			export OMPI_MCA_btl=self,sm

	mpirun -np ${SLURM_NTASKS} ./my_mpi_program</syntaxhighlight>		mpirun -np ${SLURM_NTASKS} ./my_mpi_program</syntaxhighlight>
Line 140:		Line 145:
	module load 2024 foss/2024a		module load 2024 foss/2024a

	mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \		mpirun --mca pml cm --mca mtl ofi \
			--mca mtl_ofi_provider_include psm2 \
			--mca btl self,sm \
	-x FI_PROVIDER=psm2 \		-x FI_PROVIDER=psm2 \
	-np ${SLURM_NTASKS} ./my_mpi_program ~~--input data.nc~~		-np ${SLURM_NTASKS} ./my_mpi_program
	</syntaxhighlight>		</syntaxhighlight>Note some of the settings can be done by OpenMPI's own tags.

	=== Exporting variables to the other nodes: the <code>-x</code> flag ===		=== Exporting variables to the other nodes: the <code>-x</code> flag ===

	<code>-x</code> hands a named environment variable to every rank:		<code>-x</code> hands a named environment variable to every rank, including those in other nodes :

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 155:		Line 162:
	<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.		<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.

	'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' '''Though, do not read that as universal, it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which ~~predates the PRRTE launcher and has~~ not ~~been tested here~~; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.		'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' '''Though, do not read that as universal, it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which may not propagate your environment automatically; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

	So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.		So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

Honfi001: Add recommended environment section

2026-08-06T14:46:43Z

Add recommended environment section

← Older revision		Revision as of 14:46, 6 August 2026
Line 67:		Line 67:

	If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.		If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.

			== Recommended Environment ==
			<syntaxhighlight lang="bash">
			export OMPI_MCA_pml=cm
			export OMPI_MCA_mtl=ofi
			export FI_PROVIDER=psm2
			export OMPI_MCA_btl=self,sm #substitute sm with vader for v4
			</syntaxhighlight>

	== Running on a single node ==		== Running on a single node ==

Honfi001: Trimming

2026-08-05T14:04:43Z

Trimming

← Older revision		Revision as of 14:04, 5 August 2026
Line 105:		Line 105:
	\| <code>psm2</code> \|\| Validated at line rate (~97 Gbps), on both Intel and AMD nodes, under all three toolchains \|\| '''Yes — this is the one'''		\| <code>psm2</code> \|\| Validated at line rate (~97 Gbps), on both Intel and AMD nodes, under all three toolchains \|\| '''Yes — this is the one'''
	\|-		\|-
	\| <code>opx</code> \|\| ~~Broken for inter-node traffic on OPA100: crashes either on entering rendezvous for larger messages, or during MPI start-up, depending on the libfabric version~~ \|\| No		\| <code>opx</code> \|\| Not currently operational \|\| Not yet
	\|}		\|}

	Because ~~<code>opx</code>~~ can be picked up automatically, '''an inter-node MPI job should ask for <code>psm2</code> ~~by name~~''' rather than trusting the default:		Because TCP can be picked up automatically, '''an inter-node MPI job should explicitly ask for <code>psm2</code>''' rather than trusting the default:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 136:		Line 136:
	-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc		-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
	</syntaxhighlight>		</syntaxhighlight>

	~~To check what libfabric can actually see, ask it — on a compute node, not the login node:~~

	~~<syntaxhighlight lang="bash">~~
	~~fi_info -p psm2~~
	~~</syntaxhighlight>~~

	~~If that comes back empty under a given bucket, the <code>PSM2</code> module supplies the provider; load it alongside your toolchain.~~

	=== Exporting variables to the other nodes: the <code>-x</code> flag ===		=== Exporting variables to the other nodes: the <code>-x</code> flag ===
Line 155:		Line 147:
	<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.		<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.

	'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''~~' Measured on Anunna, two nodes, OpenMPI 5.0.7, four ways:~~		'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' '''Though, do not read that as universal, it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which predates the PRRTE launcher and has not been tested here; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

	~~{\| class="wikitable"~~
	~~! What was run !! Did the variable reach the ranks?~~
	\|-
	~~\| Plain <code>export</code>, no <code>-x</code> \|\| Yes~~
	\|-
	~~\| <code>-x MY_TEST</code> \|\| Yes~~
	\|-
	~~\| Slurm's own forwarding suppressed (<code>SLURM_EXPORT_ENV=NONE</code>) \|\| Yes~~
	\|-
	~~\| Variable removed from <code>mpirun</code>'s own environment (<code>env -u</code>) \|\| '''No~~'''
	\|}

	~~Taken together those say the ranks inherit~~ '~~''<code>mpirun</code>'s''' environment~~, ~~and that OpenMPI's own process-management layer — PRRTE and PMIx — is what delivers it, not the batch system.~~

	~~'''Do~~ not read that as universal, ~~because~~ it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which predates the PRRTE launcher and has not been tested here; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

	So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.		So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

Honfi001: Trimming

2026-08-05T13:55:14Z

Trimming

← Older revision		Revision as of 13:55, 5 August 2026
Line 39:		Line 39:
	To get the full toolchain from the 2024 bucket:		To get the full toolchain from the 2024 bucket:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">module load 2024
	module load 2024		module load 2024 OpenMPI/5.0.3 # OpenMPI alone</syntaxhighlight>
	module load ~~foss~~/~~2024a~~
	</syntaxhighlight>

	Or, ~~equivalently~~, ~~on one line —~~ <code>module ~~load~~</code> ~~takes several names and loads them left to right:~~		The short version number is enough — you do '''not''' need to type the compiler suffix. <code>OpenMPI/5.0.3</code> resolves to <code>OpenMPI/5.0.3-GCC-13.3.0</code>, which is the full name <code>module list</code> will show you afterwards. If you want to see everything a bucket offers, <code>module avail OpenMPI</code> lists it once that bucket is loaded.

	~~<syntaxhighlight lang="bash">~~		Alternatively, swap the second line for modules that bundle OpenMPI with other additional modules:
	~~module load 2024 foss/2024a~~
	~~</syntaxhighlight>~~

	~~Swap the second~~ module ~~for whichever of the three routes you want:~~		<syntaxhighlight lang="bash">module load 2024 gompi/2024a # GCC + OpenMPI, no maths libraries
			module load foss/2024a # GCC + OpenMPI + maths libraries</syntaxhighlight>

	~~<syntaxhighlight lang="bash">~~		'''Check what you actually got.''' Two commands, both worth the habit
	~~module load 2024 gompi/2024a # GCC + OpenMPI, no maths libraries~~
	~~module load 2024 OpenMPI/5.0.3 # OpenMPI alone~~
	~~</syntaxhighlight>~~

	The short version number is enough — you do '''not''' need to type the compiler suffix. <code>OpenMPI/5.0.3</code> resolves to <code>OpenMPI/5.0.3-GCC-13.3.0</code>, which is the full name <code>module list</code> will show you afterwards. If you want to see everything a bucket offers, <code>module avail OpenMPI</code> lists it once that bucket is loaded.

	'''Check what you actually got.''' Two commands, both worth the habit:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 65:		Line 55:
	mpirun --version # confirms which OpenMPI is on your PATH		mpirun --version # confirms which OpenMPI is on your PATH
	</syntaxhighlight>		</syntaxhighlight>

	~~<code>module list</code> will show more than you asked for — loading a toolchain pulls in its compiler, binutils and so on. That is expected.~~

	~~If you are running software that has its own module, '''load that instead''' and let it bring the right MPI with it as a dependency:~~

	~~<syntaxhighlight lang="bash">~~
	~~module load 2024~~
	~~module load <YourSoftware>/<version> # OpenMPI arrives as a dependency~~
	~~</syntaxhighlight>~~

	Where a package exists in more than one toolchain flavour, the full name ends in the toolchain it was built with — <code>-foss-2024a</code> and so on — which tells you which OpenMPI you are getting.

	~~A few things worth knowing:~~

	* '''<code>ml</code> is shorthand''' for <code>module</code>, so <code>ml 2024 foss/2024a</code> is the same as the load above, and <code>ml</code> on its own lists what is loaded.
	* '''Load modules inside your job script, not just in your shell.''' A batch job does not reliably inherit the environment you submitted from, so the <code>module load</code> lines belong in the script itself — as they are in every example on this page.
	* '''For a clean slate, <code>module reset</code> is usually what you want''' — it returns you to the system default set rather than stripping everything. <code>module purge</code> also works, and does '''not''' cost you Slurm: the <code>slurm</code> module is ''sticky'' (Lmod marks it <code>(S)</code> and would need <code>--force</code> to remove), so it survives a purge.
	* '''Stay inside one bucket per job.''' Mixing software built with different compilers is a reliable source of mysterious failures.

	=== Which one to load ===		=== Which one to load ===
Line 98:		Line 70:
	== Running on a single node ==		== Running on a single node ==

	~~Nothing runs~~ on the login nodes — they are shared, and are for editing, submitting and light housekeeping only. Describe the job in a script, keep the data on Lustre under <code>$myScratch</code>, and hand it to <code>sbatch</code>.		Remember that nothing should run on the login nodes — they are shared, and are for editing, submitting and light housekeeping only. Describe the job in a script, keep the data on Lustre under <code>$myScratch</code>, and hand it to <code>sbatch</code>.

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">#!/bin/bash
	#!/bin/bash
	#SBATCH --job-name=mpi_single		#SBATCH --job-name=mpi_single
	#SBATCH --nodes=1		#SBATCH --nodes=1
Line 114:		Line 85:
	module load 2024 foss/2024a		module load 2024 foss/2024a

	mpirun -np ${SLURM_NTASKS} ./my_mpi_program ~~--input data.nc~~		mpirun -np ${SLURM_NTASKS} ./my_mpi_program</syntaxhighlight>
	</syntaxhighlight>

	Two habits in there are worth keeping:		Two habits in there are worth keeping:

Honfi001: Created page with "'''OpenMPI''' is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node. This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport,..."

2026-08-05T13:43:29Z

Created page with "'''OpenMPI''' is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node. This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport,..."

New page

'''OpenMPI''' is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node.

This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport, which on Anunna needs saying out loud. If you are still deciding whether MPI is the right shape for your work at all, start at [[Workflows/Multi-Process|Multi-Process Workflows]] — that page covers when to use MPI and when a [[Workflows/Embarassinly-Parallel|job array]] or [[Workflows/Multi-threaded|threads]] would serve you better. For a worked example you can run yourself, see [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]].

One thing to be clear about before anything else: '''MPI has to be built into the program.''' You cannot add it from a job script. If your software's documentation never mentions MPI or ranks, this page is not the one you need.

== What is available on Anunna ==

Software here is built with [https://easybuild.io EasyBuild] and grouped into [[Environment Modules|buckets]]. A bucket has to be loaded before you can ''load'' anything from it — and before <code>module avail</code> will list its contents. Each bucket corresponds to an EasyBuild toolchain generation:

{| class="wikitable"
! Bucket !! Toolchain !! OpenMPI !! Compiler
|-
| <code>2023</code> || <code>foss/2023a</code>, <code>gompi/2023a</code> || <code>OpenMPI/4.1.5</code> || GCC 12.3.0
|-
| <code>2024</code> || <code>foss/2024a</code>, <code>gompi/2024a</code> || <code>OpenMPI/5.0.3</code> || GCC 13.3.0
|-
| <code>2025</code> || <code>foss/2025a</code>, <code>gompi/2025a</code> || <code>OpenMPI/5.0.7</code> || GCC 14.2.0
|}

Three ways in, and they differ only in how much they bring with them:

* '''<code>foss</code>''' — the full toolchain: GCC, OpenMPI, OpenBLAS, FFTW and ScaLAPACK. Load this if your program wants linear algebra or FFTs as well as MPI.
* '''<code>gompi</code>''' — GCC and OpenMPI, nothing else. The lean choice when MPI is all you need.
* '''<code>OpenMPI/<version></code>''' — the library on its own, with its compiler underneath.

You can list them yourself with <code>module key</code>, which searches module descriptions as well as names. It reads Lmod's cache rather than your current environment, so — unlike <code>module avail</code> — it needs '''no''' bucket loaded and reports matches from every bucket at once:

<syntaxhighlight lang="bash">
module key OpenMPI # no bucket needed; searches across all of them
</syntaxhighlight>

That is the quickest way to see which OpenMPI versions exist and which toolchains carry them. Once you have picked one, ''then'' load its bucket and the module.

=== Loading it ===

Two steps, always in this order: '''the bucket, then the module.''' Nothing from a bucket can be loaded until the bucket itself is.

To get the full toolchain from the 2024 bucket:

<syntaxhighlight lang="bash">
module load 2024
module load foss/2024a
</syntaxhighlight>

Or, equivalently, on one line — <code>module load</code> takes several names and loads them left to right:

<syntaxhighlight lang="bash">
module load 2024 foss/2024a
</syntaxhighlight>

Swap the second module for whichever of the three routes you want:

<syntaxhighlight lang="bash">
module load 2024 gompi/2024a # GCC + OpenMPI, no maths libraries
module load 2024 OpenMPI/5.0.3 # OpenMPI alone
</syntaxhighlight>

The short version number is enough — you do '''not''' need to type the compiler suffix. <code>OpenMPI/5.0.3</code> resolves to <code>OpenMPI/5.0.3-GCC-13.3.0</code>, which is the full name <code>module list</code> will show you afterwards. If you want to see everything a bucket offers, <code>module avail OpenMPI</code> lists it once that bucket is loaded.

'''Check what you actually got.''' Two commands, both worth the habit:

<syntaxhighlight lang="bash">
module list # everything now loaded, including pulled-in dependencies
mpirun --version # confirms which OpenMPI is on your PATH
</syntaxhighlight>

<code>module list</code> will show more than you asked for — loading a toolchain pulls in its compiler, binutils and so on. That is expected.

If you are running software that has its own module, '''load that instead''' and let it bring the right MPI with it as a dependency:

<syntaxhighlight lang="bash">
module load 2024
module load <YourSoftware>/<version> # OpenMPI arrives as a dependency
</syntaxhighlight>

Where a package exists in more than one toolchain flavour, the full name ends in the toolchain it was built with — <code>-foss-2024a</code> and so on — which tells you which OpenMPI you are getting.

A few things worth knowing:

* '''<code>ml</code> is shorthand''' for <code>module</code>, so <code>ml 2024 foss/2024a</code> is the same as the load above, and <code>ml</code> on its own lists what is loaded.
* '''Load modules inside your job script, not just in your shell.''' A batch job does not reliably inherit the environment you submitted from, so the <code>module load</code> lines belong in the script itself — as they are in every example on this page.
* '''For a clean slate, <code>module reset</code> is usually what you want''' — it returns you to the system default set rather than stripping everything. <code>module purge</code> also works, and does '''not''' cost you Slurm: the <code>slurm</code> module is ''sticky'' (Lmod marks it <code>(S)</code> and would need <code>--force</code> to remove), so it survives a purge.
* '''Stay inside one bucket per job.''' Mixing software built with different compilers is a reliable source of mysterious failures.

=== Which one to load ===

The rule that saves the most time: '''load the same MPI the program was built against.''' Building with one MPI and running under another produces failures that look like almost anything except the mismatch causing them — hangs, garbled output, crashes inside library calls.

In practice that means:

* '''Running software from the module system?''' Load '''its''' module and let it pull in the matching MPI as a dependency. You never have to choose, and you cannot get it wrong.
* '''Running something you built yourself?''' Load the same toolchain you built it with.
* '''Building it yourself?''' Use the compiler wrappers the MPI module provides — <code>mpicc</code>, <code>mpicxx</code>, <code>mpifort</code> — which call your ordinary compiler with the MPI headers and libraries already in place.

If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.

== Running on a single node ==

Nothing runs on the login nodes — they are shared, and are for editing, submitting and light housekeeping only. Describe the job in a script, keep the data on Lustre under <code>$myScratch</code>, and hand it to <code>sbatch</code>.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=mpi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64 # 64 ranks, all on one machine
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

Two habits in there are worth keeping:

* '''<code>-np ${SLURM_NTASKS}</code>, never a hard-coded number.''' Slurm exports the task count it granted; passing that through means the request and the run cannot drift apart when you change one of them.
* '''<code>--mem-per-cpu</code>, not <code>--mem</code>.''' <code>--mem</code> is a ''per-node'' request shared out among all ranks on that node — at 64 or 128 ranks each one ends up with a few megabytes, less than an MPI process needs simply to start, and the job dies of an out-of-memory kill that points nowhere near the flag responsible.

On one node OpenMPI talks to itself through shared memory. There is nothing to configure and no network involved, which is why this case is simple and the next one is not.

== Running across several nodes ==

The moment a job spans two nodes, the ranks have to talk over the network — and on Anunna that path needs to be chosen explicitly.

Anunna's fabric is '''Omni-Path (OPA100)''', and inter-node MPI rides it through '''libfabric'''. libfabric offers more than one ''provider'' for this hardware, and on our cluster they are not equally healthy:

{| class="wikitable"
! Provider !! Status on Anunna !! Use it?
|-
| <code>psm2</code> || Validated at line rate (~97 Gbps), on both Intel and AMD nodes, under all three toolchains || '''Yes — this is the one'''
|-
| <code>opx</code> || Broken for inter-node traffic on OPA100: crashes either on entering rendezvous for larger messages, or during MPI start-up, depending on the libfabric version || No
|}

Because <code>opx</code> can be picked up automatically, '''an inter-node MPI job should ask for <code>psm2</code> by name''' rather than trusting the default:

<syntaxhighlight lang="bash">
mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \
-x FI_PROVIDER=psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

A complete multi-node job script:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=mpi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64 # 256 ranks in total
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \
-x FI_PROVIDER=psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

To check what libfabric can actually see, ask it — on a compute node, not the login node:

<syntaxhighlight lang="bash">
fi_info -p psm2
</syntaxhighlight>

If that comes back empty under a given bucket, the <code>PSM2</code> module supplies the provider; load it alongside your toolchain.

=== Exporting variables to the other nodes: the <code>-x</code> flag ===

<code>-x</code> hands a named environment variable to every rank:

<syntaxhighlight lang="bash">
mpirun -x FI_PROVIDER=psm2 -x MY_DATA_DIR -np ${SLURM_NTASKS} ./prog
</syntaxhighlight>

<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.

'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' Measured on Anunna, two nodes, OpenMPI 5.0.7, four ways:

{| class="wikitable"
! What was run !! Did the variable reach the ranks?
|-
| Plain <code>export</code>, no <code>-x</code> || Yes
|-
| <code>-x MY_TEST</code> || Yes
|-
| Slurm's own forwarding suppressed (<code>SLURM_EXPORT_ENV=NONE</code>) || Yes
|-
| Variable removed from <code>mpirun</code>'s own environment (<code>env -u</code>) || '''No'''
|}

Taken together those say the ranks inherit '''<code>mpirun</code>'s''' environment, and that OpenMPI's own process-management layer — PRRTE and PMIx — is what delivers it, not the batch system.

'''Do not read that as universal, because it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which predates the PRRTE launcher and has not been tested here; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

So treat <code>-x</code> as the '''explicit''' form rather than a workaround. It costs nothing, it states in the launch line what the run depends on, and it keeps working if the launch path changes underneath you. And if a job behaves differently across several nodes than it does on one, a variable that failed to travel is still the first thing worth suspecting.

=== A start-up gotcha ===

Do not use <code>sbatch --wrap</code> for MPI jobs. The wrapped command runs under <code>dash</code>, where Lmod's <code>module</code> function is not initialised, so module loads fail quietly and the job proceeds without the software it asked for. Write a real script starting <code>#!/bin/bash -l</code> instead.

== Advanced: pinning and rank placement ==

By default OpenMPI decides where each rank sits and which cores it may use. For most jobs that is fine. It stops being fine when a job is memory-bandwidth-bound, when ranks each run threads of their own, or when timings vary between runs for no visible reason — all of which come down to ''placement''.

Two ideas, easy to mix up:

* '''Mapping''' — which rank goes where. <code>--map-by <unit></code>, where the unit can be <code>core</code>, <code>l3cache</code>, <code>numa</code>, <code>package</code> (socket), or <code>node</code>.
* '''Binding''' — how tightly a rank is then pinned. <code>--bind-to core</code> is the usual choice; <code>--bind-to none</code> lets a rank roam.

Before tuning anything, look at what you are actually getting:

<syntaxhighlight lang="bash">
mpirun --report-bindings -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

That prints one line per rank showing the cores it is bound to. It is the only way to know rather than assume, and defaults differ between OpenMPI versions.

=== Why multiples of eight suit our AMD nodes ===

Most of Anunna's compute is AMD — '''Zen 3''' (EPYC Milan) and '''Zen 5''' (EPYC Turin). Both build a socket out of ''chiplets'' of '''8 cores''', and each chiplet has its own slice of L3 cache. Cores inside a chiplet share that cache and reach memory together; cores in different chiplets do not.

The practical consequence: '''ask for ranks in multiples of 8''', so that chiplets are filled evenly and no chiplet is left holding a single stray rank while its cache and memory channels go mostly unused. 64, 128 and 256 are comfortable numbers here; 50 or 100 are not, and will spread unevenly no matter how you map them.

Mapping by cache makes the intent explicit — one rank per chiplet, each free to use that chiplet's cores:

<syntaxhighlight lang="bash">
mpirun --map-by l3cache --bind-to core -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

A related point for memory-bound work: spreading ranks across chiplets and NUMA domains uses many memory controllers at once, while packing them close leaves most idle. On our AMD nodes that difference is large, not marginal.

=== Ranks with threads inside them ===

If each rank also runs OpenMP threads, mapping and binding stop being optional. Give each rank a block of cores with <code>PE=</code>, and bind inside it:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
mpirun --map-by ppr:8:node:PE=${SLURM_CPUS_PER_TASK} --bind-to core \
-x OMP_NUM_THREADS \
-np ${SLURM_NTASKS} ./my_hybrid_program
</syntaxhighlight>

That reads as: place 8 ranks per node, give each one <code>SLURM_CPUS_PER_TASK</code> cores, and bind them there. '''Without the <code>PE=</code> part every thread of a rank can end up on one single core''' — the job runs, produces correct results, and is many times slower than it should be. It is a silent failure, so check it with <code>--report-bindings</code>.

Note also that '''MPI binding and OpenMP affinity are separate knobs.''' <code>--map-by</code> and <code>--bind-to</code> decide which cores a ''rank'' owns; <code>OMP_PROC_BIND</code> and <code>OMP_PLACES</code> decide how that rank's ''threads'' arrange themselves within those cores. Setting the OpenMP variables cannot rescue a rank that was only given one core to begin with — the threads have nowhere to spread to. Get the MPI side right first.

== See also ==

* [[Workflows/Multi-Process|Multi-Process Workflows]] — when MPI is the right choice, and what the job script looks like.
* [[Intel MPI]] — the other MPI here, its <code>I_MPI_*</code> pinning dialect, and MKL's behaviour on AMD nodes.
* [[Workflows/Multi-threaded|Multi-threaded Workflows]] — threads on one node, for programs that are not MPI.
* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — the same calculation run serially, threaded, with MPI, and across nodes.
* [[Environment Modules]] — buckets, and how to find the module you need.
* [[Compute Hardware Overview]] — the nodes, their core counts and their memory.
* [[Batch Jobs]] — writing and submitting job scripts.
* [[Scheduler Overview (Slurm)]] — how the scheduler allocates what you ask for.