HPCwiki - Recent changes [en]

OpenMPI

Honfi001 — Thu, 06 Aug 2026 14:55:29 GMT

← Older revision		Revision as of 14:55, 6 August 2026
Line 92:		Line 92:

	module load 2024 foss/2024a		module load 2024 foss/2024a

			export OMPI_MCA_pml=cm
			export OMPI_MCA_mtl=ofi
			export FI_PROVIDER=psm2
			export OMPI_MCA_btl=self,sm

	mpirun -np ${SLURM_NTASKS} ./my_mpi_program</syntaxhighlight>		mpirun -np ${SLURM_NTASKS} ./my_mpi_program</syntaxhighlight>
Line 140:		Line 145:
	module load 2024 foss/2024a		module load 2024 foss/2024a

	mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \		mpirun --mca pml cm --mca mtl ofi \
			--mca mtl_ofi_provider_include psm2 \
			--mca btl self,sm \
	-x FI_PROVIDER=psm2 \		-x FI_PROVIDER=psm2 \
	-np ${SLURM_NTASKS} ./my_mpi_program ~~--input data.nc~~		-np ${SLURM_NTASKS} ./my_mpi_program
	</syntaxhighlight>		</syntaxhighlight>Note some of the settings can be done by OpenMPI's own tags.

	=== Exporting variables to the other nodes: the <code>-x</code> flag ===		=== Exporting variables to the other nodes: the <code>-x</code> flag ===

	<code>-x</code> hands a named environment variable to every rank:		<code>-x</code> hands a named environment variable to every rank, including those in other nodes :

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 155:		Line 162:
	<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.		<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.

	'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' '''Though, do not read that as universal, it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which ~~predates the PRRTE launcher and has~~ not ~~been tested here~~; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.		'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' '''Though, do not read that as universal, it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which may not propagate your environment automatically; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

	So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.		So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

Software

Honfi001 — Thu, 06 Aug 2026 14:48:30 GMT

← Older revision		Revision as of 14:48, 6 August 2026
Line 11:		Line 11:
	* [[Apptainer]] — containers		* [[Apptainer]] — containers
	* [[OpenMPI]]		* [[OpenMPI]]
			* [[IntelMPI]]
	* [[Licensed Software]] — software that needs a licence		* [[Licensed Software]] — software that needs a licence

OpenMPI

Honfi001 — Thu, 06 Aug 2026 14:46:43 GMT

Add recommended environment section

← Older revision		Revision as of 14:46, 6 August 2026
Line 67:		Line 67:

	If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.		If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.

			== Recommended Environment ==
			<syntaxhighlight lang="bash">
			export OMPI_MCA_pml=cm
			export OMPI_MCA_mtl=ofi
			export FI_PROVIDER=psm2
			export OMPI_MCA_btl=self,sm #substitute sm with vader for v4
			</syntaxhighlight>

	== Running on a single node ==		== Running on a single node ==

IntelMPI

Honfi001 — Wed, 05 Aug 2026 14:46:12 GMT

← Older revision		Revision as of 14:46, 5 August 2026
Line 44:		Line 44:
	module list # everything loaded, dependencies included		module list # everything loaded, dependencies included
	mpirun -version # which Intel MPI is on your PATH		mpirun -version # which Intel MPI is on your PATH
	~~</syntaxhighlight>~~

	If your software has its own module, load '''that''' instead and let it pull the matching MPI in as a dependency. A package built with this toolchain has a name ending in <code>-intel-2024a</code> or similar, which tells you what you are getting:

	~~<syntaxhighlight lang="bash">~~
	~~module load 2024~~
	~~module load <YourSoftware>/<version> # Intel MPI arrives as a dependency~~
	</syntaxhighlight>		</syntaxhighlight>

Line 138:		Line 131:
	'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.		'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.

	== Intel MPI, MKL, and ~~our~~ AMD nodes ==		== Intel MPI, MKL, and AMD nodes ==

	Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.		Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.
Line 151:		Line 144:
	</syntaxhighlight>		</syntaxhighlight>

	Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound ~~Level-3~~ work''' ~~such as <code>DGEMM</code>~~. It does nothing when MKL is only handling bandwidth-bound ~~Level-1/2 vector kernels~~, and nothing at all if your library does its own arithmetic — PETSc~~, for instance, performs its sparse matrix–vector products itself, so MKL never sees the dominant cost of a Krylov solve~~.		Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound work'''. It does nothing when MKL is only handling bandwidth-bound work, and nothing at all if your library does its own arithmetic e.g. PETSc.

	<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.		<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.

IntelMPI

Honfi001 — Wed, 05 Aug 2026 14:40:48 GMT

Created page with "'''Intel MPI''' is the other MPI implementation available on Anunna, alongside OpenMPI. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL). Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on Multi-Process Workflows. This page is the Intel-sp..."

New page

'''Intel MPI''' is the other MPI implementation available on Anunna, alongside [[OpenMPI]]. It arrives as part of the Intel toolchain, together with the Intel compilers and the Intel Math Kernel Library (MKL).

Everything general about MPI — that it runs one program as many cooperating processes, that it has to be built into the program, and when it is the right tool at all — is covered on [[Workflows/Multi-Process|Multi-Process Workflows]]. This page is the Intel-specific practice: which modules exist, how to launch, how the pinning controls differ from OpenMPI's, and one thing that catches people out on our AMD nodes.

'''Which should you use?''' Whichever your program was built with. That is not a dodge — mixing MPI implementations between build and run does not work, so in practice the software chooses for you. Where you genuinely have a choice, Intel MPI and MKL are the natural pairing for code that leans on Intel's maths libraries, and [[OpenMPI]] is the default for everything else here.

== What is available on Anunna ==

Software is built with [https://easybuild.io EasyBuild] and grouped into [[Environment Modules|buckets]]; a bucket has to be loaded before you can load anything from it. The Intel toolchain follows the same yearly generations as <code>foss</code>:

{| class="wikitable"
! Bucket !! Toolchain !! Contains
|-
| <code>2023</code> || <code>intel/2023a</code> || Intel compilers, Intel MPI, MKL
|-
| <code>2024</code> || <code>intel/2024a</code> || Intel compilers, Intel MPI, MKL
|-
| <code>2025</code> || <code>intel/2025a</code> || Intel compilers, Intel MPI, MKL
|}

<code>intel/<year>a</code> is the counterpart of <code>foss/<year>a</code>: one module that brings the whole stack — compilers, MPI, MKL and FFTW. There are leaner ways in, mirroring <code>gompi</code> on the <code>foss</code> side:

* '''<code>iimpi/<year>a</code>''' — Intel compilers and Intel MPI, without MKL. The lean choice when you only need MPI.
* '''<code>impi/<version></code>''' — the MPI library alone. In the 2025 bucket this is <code>impi/2021.15.0-intel-compilers-2025.1.1</code>; note that Intel MPI's own version numbering (2021.x) does not track the toolchain year.

To see what a bucket actually offers, ask the module system — <code>module key</code> reads Lmod's cache, so it needs no bucket loaded and searches every bucket at once:

<syntaxhighlight lang="bash">
module key intel # the toolchains
module key impi # the MPI library on its own
</syntaxhighlight>

=== Loading it ===

Bucket first, then the module — the same two steps as any other software here:

<syntaxhighlight lang="bash">
module load 2024 intel/2024a
</syntaxhighlight>

Then confirm what you got, which is worth the habit:

<syntaxhighlight lang="bash">
module list # everything loaded, dependencies included
mpirun -version # which Intel MPI is on your PATH
</syntaxhighlight>

If your software has its own module, load '''that''' instead and let it pull the matching MPI in as a dependency. A package built with this toolchain has a name ending in <code>-intel-2024a</code> or similar, which tells you what you are getting:

<syntaxhighlight lang="bash">
module load 2024
module load <YourSoftware>/<version> # Intel MPI arrives as a dependency
</syntaxhighlight>

Keep a job inside one bucket. Do not load <code>intel</code> and <code>foss</code> in the same job — they provide competing compilers, MPI libraries and BLAS, and the result is unpredictable rather than merely slow. If you want a clean starting point, <code>module reset</code> returns you to the system defaults; the <code>slurm</code> module is ''sticky'' and survives it either way.

== Running on a single node ==

Nothing runs on the login nodes. Put the work in a script, keep data on Lustre under <code>$myScratch</code>, and submit with <code>sbatch</code>.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=impi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

Two things carried over from [[OpenMPI]], for the same reasons: pass <code>${SLURM_NTASKS}</code> rather than hard-coding the rank count, and use <code>--mem-per-cpu</code> rather than <code>--mem</code>, which is a per-''node'' request divided among all the ranks on that node.

Note the <code>#!/bin/bash -l</code>. A login shell is what initialises Lmod, so without it <code>module load</code> can fail quietly. For the same reason, never launch an MPI job with <code>sbatch --wrap</code> — that runs under <code>dash</code>, where <code>module</code> does not exist.

On one node the ranks talk through shared memory and there is nothing to configure.

== Running across several nodes ==

Anunna's fabric is '''Omni-Path (OPA100)''', and both MPI implementations reach it through '''libfabric'''. The provider that works reliably here is '''<code>psm2</code>'''; the <code>opx</code> provider is broken for inter-node traffic on this hardware. So an inter-node job should name the provider rather than trust auto-selection:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=impi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 intel/2024a

mpirun -genv I_MPI_OFI_PROVIDER psm2 -genv FI_PROVIDER psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

To see what was actually selected — provider, pinning, rank layout — turn up Intel MPI's own diagnostics:

<syntaxhighlight lang="bash">
export I_MPI_DEBUG=4
</syntaxhighlight>

That prints the chosen fabric and the rank-to-core map at start-up, and is the fastest way to confirm a setting took effect instead of assuming it did.

=== Getting variables to the other nodes ===

Ranks on remote nodes are started fresh, so any variable your program or the fabric depends on has to reach them. Intel MPI's controls are:

{| class="wikitable"
! Option !! Effect
|-
| <code>-genv <VAR> <value></code> || Set one variable for '''all''' ranks. The explicit, always-safe form.
|-
| <code>-genvall</code> || Pass the whole launching environment to all ranks. This is the default.
|-
| <code>-genvlist <a,b,c></code> || Pass only the named variables.
|-
| <code>-genvnone</code> || Pass nothing.
|}

In practice '''a plain <code>export</code> in your job script does reach every rank''' — tested on Anunna across two nodes.

It arrives whatever you do to stop it: restricting Intel MPI's own propagation with <code>-genvlist PATH</code>, which should have excluded the test variable, delivered it to both nodes anyway. [[OpenMPI]] 5.0.7 behaved identically — with <code>-x</code>, without it, and even with Slurm's own environment forwarding suppressed.

The practical reading is that '''inside a Slurm allocation both implementations get your environment to the ranks''', by way of the process-management layer (PMIx and the launcher's job description) rather than the batch environment. On the OpenMPI side that was pinned down: removing the variable from <code>mpirun</code>'s own environment is the only thing that stopped it arriving — see [[OpenMPI#Exporting variables to the other nodes: the -x flag|the OpenMPI page]] for the full result.

None of which is worth depending on in a job script. It held for the current toolchains and not necessarily for older ones, so say what your run needs explicitly with <code>-genv</code> and the question stops mattering.

'''<code>-genv</code> is still the habit worth keeping''' for anything the run depends on, as in the script above. It does not rely on which layer happens to be propagating, it states the intent in the launch line, it survives a site or script that restricts propagation later, and it is the first thing to try if a variable does mysteriously fail to arrive.

== Intel MPI, MKL, and our AMD nodes ==

Most of Anunna's compute is AMD, and the Intel stack has a well-known wrinkle there. It is worth being precise about where it lives: '''the issue is in MKL, not in Intel MPI.''' But MKL arrives with the same <code>intel/<year>a</code> module, and "our Intel build is slow on the AMD nodes" is usually this, so it belongs on this page.

MKL checks the '''CPU vendor''' at run time. On a non-Intel CPU, kernels without specific Zen coverage can fall back to an '''SSE''' code path instead of using '''AVX2''', leaving much of the vector width unused. This is still the behaviour in MKL 2025.x — it has not been quietly fixed. The old <code>MKL_DEBUG_CPU_TYPE</code> workaround is gone, removed back in MKL 2020 Update 1, so anything you read recommending it is out of date.

The current mitigation is a small shim, '''<code>libfakeintel.so</code>''', <code>LD_PRELOAD</code>ed ahead of MKL. It overrides the vendor test (<code>mkl_serv_intel_cpu_true</code>) so that it returns true, and MKL then dispatches its AVX2 kernels:

<syntaxhighlight lang="bash">
export LD_PRELOAD=/path/to/libfakeintel.so
mpirun -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

Before assuming this is your problem, two things are worth checking, because often it is not. The shim only ''widens arithmetic'', so it pays off when MKL is doing '''compute-bound Level-3 work''' such as <code>DGEMM</code>. It does nothing when MKL is only handling bandwidth-bound Level-1/2 vector kernels, and nothing at all if your library does its own arithmetic — PETSc, for instance, performs its sparse matrix–vector products itself, so MKL never sees the dominant cost of a Krylov solve.

<code>MKL_VERBOSE=1</code> is the quick way to find out: it prints every MKL call, so you can see whether MKL is on your hot path at all and what it is being asked to do. Keep it off any run you intend to time — at high call counts the logging is itself a real cost.

If MKL turns out to suit your workload poorly on the AMD nodes, the <code>foss</code> toolchain's OpenBLAS-based BLAS is worth benchmarking against.

== Advanced: pinning and rank placement ==

Intel MPI and OpenMPI express placement differently: OpenMPI takes command-line flags, Intel MPI reads <code>I_MPI_*</code> environment variables. The intents map across cleanly, so if you know one dialect this table gives you the other:

{| class="wikitable"
! Intent !! Intel MPI !! OpenMPI
|-
| One rank per physical core, no migration || <code>I_MPI_PIN=1</code>, <code>I_MPI_PIN_DOMAIN=core</code> || <code>--bind-to core</code>
|-
| Fill the node, neighbours adjacent || <code>I_MPI_PIN_ORDER=compact</code> || <code>--map-by core</code>
|-
| Under-subscribed, spread across all NUMA domains || <code>I_MPI_PIN_ORDER=scatter</code> || <code>--map-by numa --bind-to core</code>
|-
| One rank per chiplet / L3 slice || <code>I_MPI_PIN_DOMAIN=cache3</code> || <code>--map-by l3cache --bind-to core</code>
|-
| Print the map actually used || <code>I_MPI_DEBUG=4</code> || <code>--report-bindings</code>
|}

Three rules that matter more than the individual settings:

* '''Verify, do not assume.''' Print the map with <code>I_MPI_DEBUG=4</code> before and after any change. Placement bugs do not announce themselves; the job runs and is simply slower than it should be.
* '''One source of truth.''' Pin with the MPI launcher '''or''' with Slurm (<code>--cpu-bind</code>, <code>--distribution</code>), never both. Two pinners fighting each other produce a nonsense map.
* '''Spread suits memory-bound work.''' AMD sockets are built from 8-core chiplets, each with its own cache slice and share of the memory channels. A bandwidth-bound job saturates memory well before every core is busy, so spreading a reduced number of ranks across all NUMA domains keeps every memory channel active, while packing them together leaves most idle. Asking for ranks in multiples of 8 keeps chiplets evenly filled.

For the reasoning behind that last point, and the equivalent OpenMPI syntax, see [[OpenMPI#Advanced: pinning and rank placement|the OpenMPI page]].

== See also ==

* [[OpenMPI]] — the other MPI here, and the default for most software.
* [[Workflows/Multi-Process|Multi-Process Workflows]] — whether MPI is the right shape for your work.
* [[Environment Modules]] — buckets, and finding the module you need.
* [[Compute Hardware Overview]] — what the nodes actually are.
* [[Batch Jobs]] — writing and submitting job scripts.
* [[Scheduler Overview (Slurm)]] — how the scheduler allocates what you ask for.

Software

Honfi001 — Wed, 05 Aug 2026 14:07:15 GMT

← Older revision		Revision as of 14:07, 5 August 2026
Line 10:		Line 10:
	* [[Julia]] — Julia		* [[Julia]] — Julia
	* [[Apptainer]] — containers		* [[Apptainer]] — containers
			* [[OpenMPI]]
	* [[Licensed Software]] — software that needs a licence		* [[Licensed Software]] — software that needs a licence

OpenMPI

Honfi001 — Wed, 05 Aug 2026 14:04:43 GMT

Trimming

Show changes

OpenMPI

Honfi001 — Wed, 05 Aug 2026 13:43:29 GMT

Created page with "'''OpenMPI''' is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node. This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport,..."

New page

'''OpenMPI''' is the MPI library Anunna is built around. MPI — the Message Passing Interface — is how a single program runs as many cooperating processes at once, each with its own private memory, passing messages between themselves as the calculation proceeds. It is what lets one job use more than one node.

This page is the practical side: which OpenMPI modules exist here, how to launch a job on one node and across several, and how to choose the network transport, which on Anunna needs saying out loud. If you are still deciding whether MPI is the right shape for your work at all, start at [[Workflows/Multi-Process|Multi-Process Workflows]] — that page covers when to use MPI and when a [[Workflows/Embarassinly-Parallel|job array]] or [[Workflows/Multi-threaded|threads]] would serve you better. For a worked example you can run yourself, see [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]].

One thing to be clear about before anything else: '''MPI has to be built into the program.''' You cannot add it from a job script. If your software's documentation never mentions MPI or ranks, this page is not the one you need.

== What is available on Anunna ==

Software here is built with [https://easybuild.io EasyBuild] and grouped into [[Environment Modules|buckets]]. A bucket has to be loaded before you can ''load'' anything from it — and before <code>module avail</code> will list its contents. Each bucket corresponds to an EasyBuild toolchain generation:

{| class="wikitable"
! Bucket !! Toolchain !! OpenMPI !! Compiler
|-
| <code>2023</code> || <code>foss/2023a</code>, <code>gompi/2023a</code> || <code>OpenMPI/4.1.5</code> || GCC 12.3.0
|-
| <code>2024</code> || <code>foss/2024a</code>, <code>gompi/2024a</code> || <code>OpenMPI/5.0.3</code> || GCC 13.3.0
|-
| <code>2025</code> || <code>foss/2025a</code>, <code>gompi/2025a</code> || <code>OpenMPI/5.0.7</code> || GCC 14.2.0
|}

Three ways in, and they differ only in how much they bring with them:

* '''<code>foss</code>''' — the full toolchain: GCC, OpenMPI, OpenBLAS, FFTW and ScaLAPACK. Load this if your program wants linear algebra or FFTs as well as MPI.
* '''<code>gompi</code>''' — GCC and OpenMPI, nothing else. The lean choice when MPI is all you need.
* '''<code>OpenMPI/<version></code>''' — the library on its own, with its compiler underneath.

You can list them yourself with <code>module key</code>, which searches module descriptions as well as names. It reads Lmod's cache rather than your current environment, so — unlike <code>module avail</code> — it needs '''no''' bucket loaded and reports matches from every bucket at once:

<syntaxhighlight lang="bash">
module key OpenMPI # no bucket needed; searches across all of them
</syntaxhighlight>

That is the quickest way to see which OpenMPI versions exist and which toolchains carry them. Once you have picked one, ''then'' load its bucket and the module.

=== Loading it ===

Two steps, always in this order: '''the bucket, then the module.''' Nothing from a bucket can be loaded until the bucket itself is.

To get the full toolchain from the 2024 bucket:

<syntaxhighlight lang="bash">
module load 2024
module load foss/2024a
</syntaxhighlight>

Or, equivalently, on one line — <code>module load</code> takes several names and loads them left to right:

<syntaxhighlight lang="bash">
module load 2024 foss/2024a
</syntaxhighlight>

Swap the second module for whichever of the three routes you want:

<syntaxhighlight lang="bash">
module load 2024 gompi/2024a # GCC + OpenMPI, no maths libraries
module load 2024 OpenMPI/5.0.3 # OpenMPI alone
</syntaxhighlight>

The short version number is enough — you do '''not''' need to type the compiler suffix. <code>OpenMPI/5.0.3</code> resolves to <code>OpenMPI/5.0.3-GCC-13.3.0</code>, which is the full name <code>module list</code> will show you afterwards. If you want to see everything a bucket offers, <code>module avail OpenMPI</code> lists it once that bucket is loaded.

'''Check what you actually got.''' Two commands, both worth the habit:

<syntaxhighlight lang="bash">
module list # everything now loaded, including pulled-in dependencies
mpirun --version # confirms which OpenMPI is on your PATH
</syntaxhighlight>

<code>module list</code> will show more than you asked for — loading a toolchain pulls in its compiler, binutils and so on. That is expected.

If you are running software that has its own module, '''load that instead''' and let it bring the right MPI with it as a dependency:

<syntaxhighlight lang="bash">
module load 2024
module load <YourSoftware>/<version> # OpenMPI arrives as a dependency
</syntaxhighlight>

Where a package exists in more than one toolchain flavour, the full name ends in the toolchain it was built with — <code>-foss-2024a</code> and so on — which tells you which OpenMPI you are getting.

A few things worth knowing:

* '''<code>ml</code> is shorthand''' for <code>module</code>, so <code>ml 2024 foss/2024a</code> is the same as the load above, and <code>ml</code> on its own lists what is loaded.
* '''Load modules inside your job script, not just in your shell.''' A batch job does not reliably inherit the environment you submitted from, so the <code>module load</code> lines belong in the script itself — as they are in every example on this page.
* '''For a clean slate, <code>module reset</code> is usually what you want''' — it returns you to the system default set rather than stripping everything. <code>module purge</code> also works, and does '''not''' cost you Slurm: the <code>slurm</code> module is ''sticky'' (Lmod marks it <code>(S)</code> and would need <code>--force</code> to remove), so it survives a purge.
* '''Stay inside one bucket per job.''' Mixing software built with different compilers is a reliable source of mysterious failures.

=== Which one to load ===

The rule that saves the most time: '''load the same MPI the program was built against.''' Building with one MPI and running under another produces failures that look like almost anything except the mismatch causing them — hangs, garbled output, crashes inside library calls.

In practice that means:

* '''Running software from the module system?''' Load '''its''' module and let it pull in the matching MPI as a dependency. You never have to choose, and you cannot get it wrong.
* '''Running something you built yourself?''' Load the same toolchain you built it with.
* '''Building it yourself?''' Use the compiler wrappers the MPI module provides — <code>mpicc</code>, <code>mpicxx</code>, <code>mpifort</code> — which call your ordinary compiler with the MPI headers and libraries already in place.

If none of those settles it, prefer the newest bucket your software is available in: the OpenMPI it carries is the one that has had the most attention here.

== Running on a single node ==

Nothing runs on the login nodes — they are shared, and are for editing, submitting and light housekeeping only. Describe the job in a script, keep the data on Lustre under <code>$myScratch</code>, and hand it to <code>sbatch</code>.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=mpi_single
#SBATCH --nodes=1
#SBATCH --ntasks=64 # 64 ranks, all on one machine
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G # per CORE, not per node
#SBATCH --time=01:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

Two habits in there are worth keeping:

* '''<code>-np ${SLURM_NTASKS}</code>, never a hard-coded number.''' Slurm exports the task count it granted; passing that through means the request and the run cannot drift apart when you change one of them.
* '''<code>--mem-per-cpu</code>, not <code>--mem</code>.''' <code>--mem</code> is a ''per-node'' request shared out among all ranks on that node — at 64 or 128 ranks each one ends up with a few megabytes, less than an MPI process needs simply to start, and the job dies of an out-of-memory kill that points nowhere near the flag responsible.

On one node OpenMPI talks to itself through shared memory. There is nothing to configure and no network involved, which is why this case is simple and the next one is not.

== Running across several nodes ==

The moment a job spans two nodes, the ranks have to talk over the network — and on Anunna that path needs to be chosen explicitly.

Anunna's fabric is '''Omni-Path (OPA100)''', and inter-node MPI rides it through '''libfabric'''. libfabric offers more than one ''provider'' for this hardware, and on our cluster they are not equally healthy:

{| class="wikitable"
! Provider !! Status on Anunna !! Use it?
|-
| <code>psm2</code> || Validated at line rate (~97 Gbps), on both Intel and AMD nodes, under all three toolchains || '''Yes — this is the one'''
|-
| <code>opx</code> || Broken for inter-node traffic on OPA100: crashes either on entering rendezvous for larger messages, or during MPI start-up, depending on the libfabric version || No
|}

Because <code>opx</code> can be picked up automatically, '''an inter-node MPI job should ask for <code>psm2</code> by name''' rather than trusting the default:

<syntaxhighlight lang="bash">
mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \
-x FI_PROVIDER=psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

A complete multi-node job script:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=mpi_multi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64 # 256 ranks in total
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --time=02:00:00
#SBATCH --output=%x-%j.out

cd "$myScratch/my_simulation"

module load 2024 foss/2024a

mpirun --mca pml cm --mca mtl ofi --mca mtl_ofi_provider_include psm2 \
-x FI_PROVIDER=psm2 \
-np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

To check what libfabric can actually see, ask it — on a compute node, not the login node:

<syntaxhighlight lang="bash">
fi_info -p psm2
</syntaxhighlight>

If that comes back empty under a given bucket, the <code>PSM2</code> module supplies the provider; load it alongside your toolchain.

=== Exporting variables to the other nodes: the <code>-x</code> flag ===

<code>-x</code> hands a named environment variable to every rank:

<syntaxhighlight lang="bash">
mpirun -x FI_PROVIDER=psm2 -x MY_DATA_DIR -np ${SLURM_NTASKS} ./prog
</syntaxhighlight>

<code>-x VAR=value</code> sets it; <code>-x VAR</code> passes through whatever value it already has. Use one <code>-x</code> per variable.

'''With OpenMPI 5 inside a Slurm job, you usually do not need it.''' Measured on Anunna, two nodes, OpenMPI 5.0.7, four ways:

{| class="wikitable"
! What was run !! Did the variable reach the ranks?
|-
| Plain <code>export</code>, no <code>-x</code> || Yes
|-
| <code>-x MY_TEST</code> || Yes
|-
| Slurm's own forwarding suppressed (<code>SLURM_EXPORT_ENV=NONE</code>) || Yes
|-
| Variable removed from <code>mpirun</code>'s own environment (<code>env -u</code>) || '''No'''
|}

Taken together those say the ranks inherit '''<code>mpirun</code>'s''' environment, and that OpenMPI's own process-management layer — PRRTE and PMIx — is what delivers it, not the batch system.

'''Do not read that as universal, because it depends on the version.''' It was verified for OpenMPI 5.0.7, in the <code>2025</code> bucket. The <code>2023</code> bucket carries '''OpenMPI 4.1.5''', which predates the PRRTE launcher and has not been tested here; historically variables did have to be passed by hand. And in a hostfile launch over <code>ssh</code>, rather than inside a Slurm allocation, remote shells start clean and nothing travels at all.

So use <code>-x</code> if you are on the <code>2023</code> bucket, if you are launching outside Slurm, or if you simply would rather not depend on any of the above. It costs nothing when it turns out to be redundant, and it is the one form that works in every case.

So treat <code>-x</code> as the '''explicit''' form rather than a workaround. It costs nothing, it states in the launch line what the run depends on, and it keeps working if the launch path changes underneath you. And if a job behaves differently across several nodes than it does on one, a variable that failed to travel is still the first thing worth suspecting.

=== A start-up gotcha ===

Do not use <code>sbatch --wrap</code> for MPI jobs. The wrapped command runs under <code>dash</code>, where Lmod's <code>module</code> function is not initialised, so module loads fail quietly and the job proceeds without the software it asked for. Write a real script starting <code>#!/bin/bash -l</code> instead.

== Advanced: pinning and rank placement ==

By default OpenMPI decides where each rank sits and which cores it may use. For most jobs that is fine. It stops being fine when a job is memory-bandwidth-bound, when ranks each run threads of their own, or when timings vary between runs for no visible reason — all of which come down to ''placement''.

Two ideas, easy to mix up:

* '''Mapping''' — which rank goes where. <code>--map-by <unit></code>, where the unit can be <code>core</code>, <code>l3cache</code>, <code>numa</code>, <code>package</code> (socket), or <code>node</code>.
* '''Binding''' — how tightly a rank is then pinned. <code>--bind-to core</code> is the usual choice; <code>--bind-to none</code> lets a rank roam.

Before tuning anything, look at what you are actually getting:

<syntaxhighlight lang="bash">
mpirun --report-bindings -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

That prints one line per rank showing the cores it is bound to. It is the only way to know rather than assume, and defaults differ between OpenMPI versions.

=== Why multiples of eight suit our AMD nodes ===

Most of Anunna's compute is AMD — '''Zen 3''' (EPYC Milan) and '''Zen 5''' (EPYC Turin). Both build a socket out of ''chiplets'' of '''8 cores''', and each chiplet has its own slice of L3 cache. Cores inside a chiplet share that cache and reach memory together; cores in different chiplets do not.

The practical consequence: '''ask for ranks in multiples of 8''', so that chiplets are filled evenly and no chiplet is left holding a single stray rank while its cache and memory channels go mostly unused. 64, 128 and 256 are comfortable numbers here; 50 or 100 are not, and will spread unevenly no matter how you map them.

Mapping by cache makes the intent explicit — one rank per chiplet, each free to use that chiplet's cores:

<syntaxhighlight lang="bash">
mpirun --map-by l3cache --bind-to core -np ${SLURM_NTASKS} ./my_mpi_program
</syntaxhighlight>

A related point for memory-bound work: spreading ranks across chiplets and NUMA domains uses many memory controllers at once, while packing them close leaves most idle. On our AMD nodes that difference is large, not marginal.

=== Ranks with threads inside them ===

If each rank also runs OpenMP threads, mapping and binding stop being optional. Give each rank a block of cores with <code>PE=</code>, and bind inside it:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
mpirun --map-by ppr:8:node:PE=${SLURM_CPUS_PER_TASK} --bind-to core \
-x OMP_NUM_THREADS \
-np ${SLURM_NTASKS} ./my_hybrid_program
</syntaxhighlight>

That reads as: place 8 ranks per node, give each one <code>SLURM_CPUS_PER_TASK</code> cores, and bind them there. '''Without the <code>PE=</code> part every thread of a rank can end up on one single core''' — the job runs, produces correct results, and is many times slower than it should be. It is a silent failure, so check it with <code>--report-bindings</code>.

Note also that '''MPI binding and OpenMP affinity are separate knobs.''' <code>--map-by</code> and <code>--bind-to</code> decide which cores a ''rank'' owns; <code>OMP_PROC_BIND</code> and <code>OMP_PLACES</code> decide how that rank's ''threads'' arrange themselves within those cores. Setting the OpenMP variables cannot rescue a rank that was only given one core to begin with — the threads have nowhere to spread to. Get the MPI side right first.

== See also ==

* [[Workflows/Multi-Process|Multi-Process Workflows]] — when MPI is the right choice, and what the job script looks like.
* [[Intel MPI]] — the other MPI here, its <code>I_MPI_*</code> pinning dialect, and MKL's behaviour on AMD nodes.
* [[Workflows/Multi-threaded|Multi-threaded Workflows]] — threads on one node, for programs that are not MPI.
* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — the same calculation run serially, threaded, with MPI, and across nodes.
* [[Environment Modules]] — buckets, and how to find the module you need.
* [[Compute Hardware Overview]] — the nodes, their core counts and their memory.
* [[Batch Jobs]] — writing and submitting job scripts.
* [[Scheduler Overview (Slurm)]] — how the scheduler allocates what you ask for.

Software

Honfi001 — Wed, 05 Aug 2026 11:36:12 GMT

← Older revision		Revision as of 11:36, 5 August 2026
Line 7:		Line 7:
	* [[Installing Personal Software]] — installing into your own space		* [[Installing Personal Software]] — installing into your own space
	* [[Python]] — Python (modules, Miniforge, virtual environments)		* [[Python]] — Python (modules, Miniforge, virtual environments)
	* [[R]] — R and R libraries		* [[R]] — R-lang and libraries
	* [[Julia]] — Julia		* [[Julia]] — Julia
	* [[Apptainer]] — containers		* [[Apptainer]] — containers
	* [[Licensed Software]] — software that needs a licence		* [[Licensed Software]] — software that needs a licence

Workflows

Honfi001 — Wed, 05 Aug 2026 09:55:42 GMT

← Older revision		Revision as of 09:55, 5 August 2026
Line 16:		Line 16:
	* [[Workflows/Embarassinly-Parallel\|Embarrassingly Parallel]]		* [[Workflows/Embarassinly-Parallel\|Embarrassingly Parallel]]
	* [[Workflows/Multi-threaded\|Multi-threaded]]		* [[Workflows/Multi-threaded\|Multi-threaded]]
	* Multi-Process ~~Workflows~~		* [[Workflows/Multi-Process\|Multi-Process]]
	* Hybrid Workflows		* Hybrid Workflows
	* GPU workflows		* GPU workflows

Workflows/Multi-Process

Honfi001 — Wed, 05 Aug 2026 09:54:26 GMT

Created page with "Multi-threaded work gave one process many arms, and then ran into a wall: threads share memory, and shared memory stops at the edge of the machine. This page is how a single calculation gets past that wall — not by growing one process, but by running '''many separate processes''', each with its own private memory, that send messages to each other as the calculation proceeds. The standard way to do that is '''MPI''', the Message Passing Inte..."

New page

[[Workflows/Multi-threaded|Multi-threaded]] work gave one process many arms, and then ran into a wall: threads share memory, and shared memory stops at the edge of the machine. This page is how a single calculation gets past that wall — not by growing one process, but by running '''many separate processes''', each with its own private memory, that send messages to each other as the calculation proceeds. The standard way to do that is '''MPI''', the Message Passing Interface, and it is what lets one job span several nodes of the cluster at once.

This is also the first rung where the '''program''' has to have been built for it. On the previous two rungs you changed the job script and the software went along with it. Here the parallelism lives inside the source code, and no <code>#SBATCH</code> line can put it there. So the first question is not how to write the script — it is whether what you are running is an MPI program at all.

== When this is you ==

You are looking at a multi-process workflow when the software was written to run as many cooperating copies of itself:

* Its documentation mentions '''MPI''', or tells you to start it with <code>mpirun</code>, <code>mpiexec</code>, or <code>srun</code> rather than by name.
* It talks about '''ranks''' or '''processes''' — and about running across '''nodes''' — where a threaded tool would talk about threads.
* Installing or loading it pulled in an MPI library: a module with <code>OpenMPI</code> or <code>impi</code> in its name, or a toolchain module such as <code>foss</code> that contains one.

Large simulation codes are the usual inhabitants of this rung — molecular dynamics and computational fluid dynamics especially, with GROMACS and LAMMPS the ones people meet first — along with some assemblers and phylogenetics packages that ship a separate MPI build alongside the ordinary one.

Two tells that you are '''not''' here. If the program only offers a <code>--threads</code>-style option, it is [[Workflows/Multi-threaded|multi-threaded]] and one node is its limit; asking Slurm for tasks it does not know about will not change that. And if what you have is many runs rather than one big one, you want [[Workflows/Embarassinly-Parallel|a job array]]. That distinction is worth being blunt about, because "many processes" describes both: an array also runs many processes at once, but its tasks never speak to each other, while MPI ranks speak constantly. If your runs are genuinely independent, take the array — it is simpler, it starts sooner, and it does not require an MPI program.

== What it looks like on Anunna ==

Everything from the [[Workflows/Serial|serial]] job still holds: nothing runs on the login node, work lives under <code>$myScratch</code> on Lustre rather than in your home directory, and the job is a short script handed to <code>sbatch</code>. What changes is which resource lines you raise, and the line that starts the program.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=my_simulation
#SBATCH --nodes=2 # two machines ...
#SBATCH --ntasks-per-node=64 # ... 64 processes on each: 128 ranks in total
#SBATCH --cpus-per-task=1 # each rank single-threaded
#SBATCH --mem-per-cpu=2G # memory PER CORE, not per node -- see below
#SBATCH --time=01:00:00 # wall-clock limit; the job is killed past this
#SBATCH --output=%x-%j.out # stdout + stderr, named by job name and id

cd "$myScratch/my_simulation"

module load 2024 foss/2024a # GCC + OpenMPI; match how the program was built

mpirun -np ${SLURM_NTASKS} ./my_mpi_program --input data.nc
</syntaxhighlight>

Set against the serial script, the interesting part is which number went up:

* '''<code>--ntasks</code> is what grew''' — expressed here as <code>--nodes</code> × <code>--ntasks-per-node</code>, which comes to 128 — while '''<code>--cpus-per-task</code> stayed at 1'''. That is the exact mirror image of the multi-threaded page, which raised <code>--cpus-per-task</code> and left <code>--ntasks</code> at 1. Those two lines are the whole difference between the two rungs: one process with many cores, or many processes with one core each.
* '''<code>--nodes</code> appears for the first time in this section.''' It can, because ranks do not share memory and therefore do not have to share a machine. You may also simply write <code>--ntasks=128</code> and let Slurm place the ranks wherever it likes; spelling out nodes and tasks-per-node instead gives you predictable placement, which makes timings comparable between runs.
* '''The launch line hands Slurm's number to the program.''' Slurm exports the task count as <code>SLURM_NTASKS</code>, and <code>mpirun -np ${SLURM_NTASKS}</code> starts exactly that many ranks. This is the same habit as <code>OMP_NUM_THREADS</code> one rung down: never hard-code the number in two places, or the request and the run will eventually disagree.

=== Which MPI module ===

The rule that matters: '''load the same MPI the program was built against.''' Mixing them — building with one MPI and running with another — produces errors that look like almost anything except the mismatch causing them.

If the software came from the module system, loading '''its''' module is the reliable route, because that pulls in the matching MPI as a dependency and you never choose by hand. If you are running something you built yourself, load the toolchain you built it with — and if you are the one doing the building, that is what the MPI module's compiler wrappers (<code>mpicc</code>, <code>mpicxx</code>, <code>mpifort</code>) are for: they call your ordinary compiler with the MPI headers and libraries already in place. The π tutorial's MPI rung uses:

<syntaxhighlight lang="bash">
module purge
module load 2024 foss/2024a # GCC 13.3 with OpenMPI 5.0.3
</syntaxhighlight>

Modules on Anunna sit inside year buckets, and a bucket has to be loaded before its contents are visible — so <code>module avail OpenMPI</code> will show you nothing useful until you have loaded a <code>2024</code> or <code>2025</code> first. Newer MPI versions live in the newer buckets. See [[Environment Modules]] for how the buckets work. Note also that <code>module purge</code> removes the <code>slurm</code> module along with everything else; reload it if your script needs Slurm commands after purging.

=== mpirun or srun ===

Both will start an MPI program, and you will meet both in documentation. The material in this section uses <code>mpirun</code> consistently, for a practical reason: it is OpenMPI's own launcher, so OpenMPI's placement options behave as its documentation says they do, whereas launching through <code>srun</code> adds a second layer that can quietly override them. Unless you have a reason to prefer <code>srun</code>, use <code>mpirun -np ${SLURM_NTASKS}</code> and stay with it.

== Memory is counted per rank now ==

On the serial page, <code>--mem=4G</code> meant four gigabytes for the job's one process. It does not mean anything like that here, and this is the mistake most likely to kill an otherwise correct MPI job.

<code>--mem</code> is a request '''per node''', and it is shared out among every rank running on that node. Ask for <code>--mem=1G</code> and put 128 ranks on the node, and each rank gets roughly eight megabytes — less than an MPI process needs merely to start up, before your data is anywhere in the picture. The job dies of an out-of-memory kill early in the run, and nothing in the error message points back at the flag responsible. This is not a hypothetical: it is precisely what happened while the π tutorial's MPI rungs were being built, and switching to <code>--mem-per-cpu</code> was the fix.

So on this rung, think per rank and write <code>--mem-per-cpu</code>. Multiply in your head to check the total is sane — <code>--mem-per-cpu=2G</code> with 64 ranks on a node is 128 GB from that node, which is comfortable on Anunna's larger nodes and impossible on its smaller ones.

== What it buys you ==

The [[Tutorials/Parallelism-Estimating-Pi|π tutorial]] runs the same ten-billion-dart estimate as an MPI program, on one node and across four, alongside the serial baseline:

<syntaxhighlight lang="text">
variant nodes cores wall_s
serial 1 1 26.008
mpi 1 128 0.555
mpi 4 64 1.090
</syntaxhighlight>

A hundred and twenty-eight ranks turn 26 seconds into about half a second. The more instructive pair is the second and third rows, which look at first glance like a penalty for leaving the node — until you notice the four-node run used '''64''' cores against the single node's '''128'''. Compared like with like, the tutorial's 64-core runs took about 1.11 s on one node and about 1.09 s spread across four: the same, within noise. For this workload, crossing the network cost nothing measurable.

That result is genuinely good news and should still be read carefully, because this problem barely communicates — the ranks work alone and exchange one number each at the very end. A real simulation exchanges data at every step, and that cost grows as ranks are added, which is why real codes have a scaling sweet spot rather than a straight line and why the only trustworthy answer comes from measuring '''your''' program on '''your''' data. The tutorial has a wrinkle in the same spirit: its single-rank MPI run takes about 70 seconds where the single-thread OpenMP run takes 26, a property of how that particular program starts up rather than a rule about MPI. Small scale is where MPI's overheads are most visible and least repaid.

== The honest limitation ==

'''The program must have been written for MPI, and you cannot bolt it on.''' No <code>#SBATCH</code> line, no module, no wrapper script adds MPI to software that does not already have it. Run a non-MPI program under <code>mpirun -np 128</code> and you do not get a parallel run; you get 128 identical copies of the same serial calculation, each ignorant of the others, quite possibly writing over each other's output files. It looks busy and it is worse than useless.

If the documentation never mentions MPI or ranks, that is your answer, and it is a perfectly ordinary one — most software is not MPI software. Go back a rung to [[Workflows/Multi-threaded|multi-threaded]], or sideways to [[Workflows/Embarassinly-Parallel|a job array]] if you have many runs rather than one.

And when you do have an MPI program, more ranks is not automatically faster. Communication is real work, the queue makes you wait longer for 256 cores than for 64, and past some rank count your run stops improving and starts getting worse. Find where that happens on a short test case before committing a long job to it.

If your MPI program can also use threads inside each rank — many of the big codes can — the combination is usually faster than either alone, and it is its own page: [[Workflows/Hybrid|Hybrid Workflows]].

== See also ==

* [[Workflows/Parallel-Computing|Parallel Computing]] — the overview, and the map to the other workflow types.
* [[Workflows/Multi-threaded|Multi-Threaded Workflows]] — the previous rung, and the one-node ceiling this page breaks.
* [[Workflows/Hybrid|Hybrid Workflows]] — MPI between nodes, threads within them.
* [[Workflows/Embarassinly-Parallel|Embarrassingly Parallel]] — for many independent runs, which need no MPI at all.
* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — its MPI and multi-node rungs are the worked example for this page.
* [[Environment Modules]] — year buckets, and how to find the MPI you need.
* [[Compute Hardware Overview]] — how many cores and how much memory a node actually has.

Workflows

Honfi001 — Wed, 05 Aug 2026 09:29:03 GMT

← Older revision		Revision as of 09:29, 5 August 2026
Line 14:		Line 14:
	* [[Workflows/Parallel-Computing\|Parallel Computing]]		* [[Workflows/Parallel-Computing\|Parallel Computing]]
	* [[Workflows/Serial\|Serial]]		* [[Workflows/Serial\|Serial]]
	* Embarrassingly Parallel		* [[Workflows/Embarassinly-Parallel\|Embarrassingly Parallel]]
	* [[Workflows/Multi-threaded\|Multi-threaded]]		* [[Workflows/Multi-threaded\|Multi-threaded]]
	* Multi-Process Workflows		* Multi-Process Workflows

Workflows/Multi-threaded

Honfi001 — Wed, 05 Aug 2026 08:02:57 GMT

Fix links

← Older revision		Revision as of 08:02, 5 August 2026
Line 1:		Line 1:
	[[Workflows/Serial~~-Workflows~~\|Serial]] ran one program on one core. This page takes that same job and lets the program spread its work across '''many''' cores on the '''same''' machine — still one process, but now with several ''threads'' inside it, all sharing the same memory. It is the gentlest real step up the ladder, because for most tools someone else already did the hard part: the program already knows how to use extra cores. Your job is only to give it some, and to tell it how many.		[[Workflows/Serial\|Serial]] ran one program on one core. This page takes that same job and lets the program spread its work across '''many''' cores on the '''same''' machine — still one process, but now with several ''threads'' inside it, all sharing the same memory. It is the gentlest real step up the ladder, because for most tools someone else already did the hard part: the program already knows how to use extra cores. Your job is only to give it some, and to tell it how many.

	If you have read the serial page you already have the whole script. This one is that script with two lines touched — and seeing exactly which two is the point.		If you have read the serial page you already have the whole script. This one is that script with two lines touched — and seeing exactly which two is the point.
Line 11:		Line 11:
	* It runs as a single program — not several cooperating copies, and not across more than one node.		* It runs as a single program — not several cooperating copies, and not across more than one node.

	Plenty of everyday tools are built this way: read aligners, <code>samtools</code>, and many of the numerical libraries underneath R and Python. If a tool has a threads option and you have been leaving it at its default of 1, this page is how you turn it up correctly on Anunna. If the program has no such option, it is [[Workflows/Serial~~-Workflows~~\|serial]] — more cores will not help it, and you should go back one rung.		Plenty of everyday tools are built this way: read aligners, <code>samtools</code>, and many of the numerical libraries underneath R and Python. If a tool has a threads option and you have been leaving it at its default of 1, this page is how you turn it up correctly on Anunna. If the program has no such option, it is [[Workflows/Serial\|serial]] — more cores will not help it, and you should go back one rung.

	== What it looks like on Anunna ==		== What it looks like on Anunna ==
Line 84:		Line 84:

	* [[Workflows/Parallel-Computing\|Parallel Computing]] — the overview, and the map to the other workflow types.		* [[Workflows/Parallel-Computing\|Parallel Computing]] — the overview, and the map to the other workflow types.
	* [[Workflows/Serial~~-Workflows~~\|Serial Workflows]] — the baseline this page starts from.		* [[Workflows/Serial\|Serial Workflows]] — the baseline this page starts from.
	* [[Workflows/Multi-Process-Workflows\|Multi-Process Workflows]] — the next rung, for when one node is not enough.		* [[Workflows/Multi-Process-Workflows\|Multi-Process Workflows]] — the next rung, for when one node is not enough.
	* [[Tutorials/Parallelism-Estimating-Pi\|Parallelism: Estimating π]] — its OpenMP rung is the worked example for this page.		* [[Tutorials/Parallelism-Estimating-Pi\|Parallelism: Estimating π]] — its OpenMP rung is the worked example for this page.
	* [[Compute Hardware Overview]] — how many cores a single node actually has.		* [[Compute Hardware Overview]] — how many cores a single node actually has.

Workflows/Embarassinly-Parallel

Honfi001 — Wed, 05 Aug 2026 07:59:04 GMT

Embarassinly Parallel

Show changes

Workflows/Embarassinly-Parallel

Honfi001 — Wed, 05 Aug 2026 07:31:59 GMT

Created page with "This tutorial is a gentle, hands-on introduction to '''parallel computing on Anunna'''. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the ''same'' calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you. You do no..."

New page

This tutorial is a gentle, hands-on introduction to '''parallel computing on Anunna'''. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the ''same'' calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you.

You do not need to write any C code. The whole exercise comes down to filling in '''two''' blanks in two job scripts, launching a batch of jobs, and reading three plots.

== What you will learn ==

* How a single program can be run serially, with threads (OpenMP), with processes (MPI), across several nodes, and on a GPU.
* How Slurm ''allocates'' resources and why your program still has to be ''told'' how many to use.
* The difference between '''strong scaling''' (same work, more workers, less time) and '''weak scaling''' (more workers, more work, same time).
* Why more workers make a calculation '''faster, not more accurate'''.
* How to read scaling and speedup plots — and where their limits are.

== Before you begin ==

You should already be comfortable logging into Anunna and submitting a basic Slurm job. If you are not sure, work through the [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] first.

You also need the '''course environment loaded''', because it sets the <code>$myScratch</code> variable that the tutorial relies on. If your instructor has not told you otherwise, it is loaded for you when you log in. You can check with:

<syntaxhighlight lang="bash">
echo "$myScratch"
</syntaxhighlight>

If that prints a path under <code>/lustre/scratch/...</code>, you are ready. If it prints nothing, load the course environment before continuing.

Everything you run lives in two places:

* '''The shared tutorial''' at <code>/lustre/shared/hpcCourses/pi-tutorial</code> — read-only, the same for everyone.
* '''Your personal copy''' at <code>$myScratch/hpcCourse/pi-tutorial</code> — created the first time you launch, and where you make your edits.

== The big idea: estimating π with random darts ==

Imagine a square dartboard with a quarter-circle drawn inside it. If you throw darts at random and they land anywhere in the square with equal chance, the fraction that land ''inside'' the quarter-circle is exactly the ratio of the two areas — which works out to π/4. So if you throw '''N''' darts and '''H''' of them land inside:

<syntaxhighlight lang="text">
π ≈ 4 × H / N
</syntaxhighlight>

Throw more darts, and the estimate gets better. This is a '''Monte Carlo''' method: a calculation built out of huge numbers of independent random trials. It has one property that makes it perfect for teaching parallelism — every dart is independent, so the work splits cleanly across as many workers as you like, with almost no coordination between them.

The tutorial implements this ''same'' dart-throwing kernel six ways:

{| class="wikitable"
! Rung !! Program !! What it uses
|-
| serial || <code>pi_serial</code> || one core (the reference)
|-
| OpenMP || <code>pi_omp</code> || many threads on one node
|-
| MPI || <code>pi_mpi</code> || many processes on one node
|-
| multi-node || <code>pi_mpi</code> || processes spread across several nodes (same program)
|-
| GPU || <code>pi_gpu</code> || thousands of device threads on one GPU
|-
| hybrid || <code>pi_hybrid</code> || MPI processes, each running OpenMP threads
|}

Because the random-number generator is identical everywhere and each worker draws its own independent stream, the ''answer'' for a given number of darts and seed is the same no matter how the work is divided. Only the ''time'' changes. That is the whole point.

== Step 1: Make the tutorial modules available ==

The tutorial ships as a set of environment modules. Point Lmod at them once per session:

<syntaxhighlight lang="bash">
module use /lustre/shared/hpcCourses/pi-tutorial/modules
module avail pi-tutorial
</syntaxhighlight>

You will see three modules. They are mutually exclusive — loading one unloads the others:

* <code>pi-tutorial</code> — the CPU rungs (serial, OpenMP, MPI, hybrid).
* <code>pi-tutorial-cuda</code> — the NVIDIA GPU rung.
* <code>pi-tutorial-rocm</code> — the AMD GPU rung.

You do not need to load these by hand for the exercise — each job script loads the right one for you. The <code>module use</code> line above is enough.

== Step 2: Get your personal copy of the scripts ==

Run the launcher once:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

On this first run it does three things: it copies the job scripts into your personal space at <code>$myScratch/hpcCourse/pi-tutorial/slurm</code>, clears any old results, and submits the whole batch of jobs. Watch them with:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Most of the jobs (serial, GPU, multi-node, hybrid) are complete and will run correctly. '''Two of them — the OpenMP and the MPI jobs — contain a deliberate blank''' and will not give a useful result yet. Fixing those blanks is your job, and it is the next step.

== Step 3: Fill in the two blanks ==

When you ask Slurm for resources, Slurm sets aside the cores or tasks you requested — but it does not automatically tell your ''program'' about them. You have to pass the number along. Slurm makes that easy by exporting the counts as environment variables. Your task is to connect the two.

Open the OpenMP script in your personal copy (use <code>nano</code> if you are not comfortable with <code>vi</code>):

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multithread.slurm
</syntaxhighlight>

You will find this line:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=FIXME
</syntaxhighlight>

The script asked Slurm for cores with <code>--cpus-per-task</code>, and Slurm exported that count as <code>SLURM_CPUS_PER_TASK</code>. Tell OpenMP to use exactly that many threads:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
</syntaxhighlight>

Now the MPI script:

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multiprocess.slurm
</syntaxhighlight>

Here the blank is the number of processes to launch:

<syntaxhighlight lang="bash">
mpirun -np FIXME pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

This script asked Slurm for tasks with <code>--ntasks</code>, and Slurm exported that as <code>SLURM_NTASKS</code>. Launch one process per task:

<syntaxhighlight lang="bash">
mpirun -np ${SLURM_NTASKS} pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

That is the entire editable surface — two lines. Save both files.

== Step 4: Launch the full sweep ==

Run the launcher again:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

This time it '''keeps your edited scripts''' (it never overwrites your copy), clears the old results, and resubmits everything. The launcher sweeps a range of sizes for you automatically — OpenMP from 1 up to 128 threads, MPI from 1 up to 128 processes — plus the GPU, multi-node, and hybrid runs.

Keep an eye on the queue:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Each finished job writes one result line into <code>$myScratch/hpcCourse/pi-tutorial/results</code>. Wait until the queue is empty before moving on.

== Step 5: Analyze your results ==

The analysis script reads every result file, prints a summary table, and writes three plots. It needs matplotlib, which lives in the SciPy bundle:

<syntaxhighlight lang="bash">
module load 2024 SciPy-bundle/2024a
python /lustre/shared/hpcCourses/pi-tutorial/analyze.py $myScratch/hpcCourse/pi-tutorial/results
</syntaxhighlight>

When it finishes you will have three PNG files in your results directory:

* <code>walltime_vs_cores.png</code> — how long each run took, against the number of cores.
* <code>speedup_vs_cores.png</code> — how much faster than the serial run, against the ideal straight line.
* <code>best_per_approach.png</code> — the single fastest run of each approach, side by side.

Copy them to your laptop (for example with <code>scp</code>) to view them, or open them through the OnDemand file browser.

== Results ==

[[File:Pi-tutorial-walltime-vs-cores.png|thumb|right|360px|Wall time falls as cores are added. Note both axes are logarithmic; a straight line here means near-ideal scaling.]]
[[File:Pi-tutorial-speedup-vs-cores.png|thumb|right|360px|Speedup against the serial baseline. The dashed line is perfect (linear) speedup; real runs sit just below it.]]
[[File:Pi-tutorial-best-per-approach.png|thumb|right|360px|The fastest run of each approach. The GPU finishes the same 10-billion-dart job in a fraction of the time the CPU rungs need.]]

A correct run produces a table like the one below (trimmed; your exact times will vary by a few percent). Every run threw the same '''10 billion''' darts.

<syntaxhighlight lang="text">
variant nodes cores device wall_s pi pct_err
serial 1 1 cpu 26.008 3.1416052 0.00040
omp 1 1 cpu 26.499 3.1416052 0.00040
omp 1 16 cpu 1.663 3.1415960 0.00011
omp 1 128 cpu 0.235 3.1416075 0.00047
mpi 1 1 cpu 69.676 3.1416052 0.00040
mpi 1 128 cpu 0.555 3.1416075 0.00047
mpi 4 64 cpu 1.090 3.1416039 0.00036
hybrid 4 128 cpu 0.217 3.1416075 0.00047
gpu 1 1 AMD_Instinct_MI210 0.080 3.1415897 0.00009
gpu 1 1 NVIDIA_A100_80GB_PCIe 0.045 3.1415897 0.00009
</syntaxhighlight>

Two things are worth noticing straight away:

* The serial run took about '''26 seconds'''; the 128-thread OpenMP run took about '''0.24 seconds'''; the A100 GPU finished in about '''0.045 seconds'''. The work was identical — only the time changed.
* Look at the <code>pi</code> column. For any given total work and seed, the estimate is '''identical''' across serial, OpenMP, MPI, multi-node, and hybrid — for example every 128-core CPU run lands on 3.1416075. Splitting the darts across more workers did not change the answer, because each worker draws its own independent random stream. (The GPU differs in the last few digits because it uses far more streams; it still lands well within the expected error.)

== Discussion ==

The plots tell a tidy story: add workers, get proportionally faster, almost along the ideal line. That is real — but it is important to understand ''why'' it looks so clean, and where the limits are.

'''This problem is unusually friendly.''' Throwing darts is "embarrassingly parallel": the workers never need to talk to each other until the very end, when their hit-counts are summed. Most real scientific codes are not like this. A fluid-dynamics solver, a molecular-dynamics simulation, or a genome assembler has data dependencies and communication between workers, and that communication eventually caps how much faster you can go (this ceiling is known as '''Amdahl's law'''). Do not expect your own code to scale as cleanly as π does — this tutorial shows you the ''best'' case, not the typical one.

'''More workers buy speed, not accuracy.''' The error of a Monte Carlo estimate shrinks with the number of darts as roughly 1/√N — so to halve the error you must throw '''four times''' as many darts. Adding cores lets you throw a fixed number of darts ''sooner'', or throw ''more'' darts in the same time, but it never makes a fixed number of darts more accurate. Speed and accuracy are separate knobs.

'''Compare equal core counts.''' In the table, the 4-node MPI run (1.09 s) looks ''slower'' than the 1-node MPI run (0.56 s). It is not a penalty for crossing nodes — the 4-node run used '''64''' cores while the 1-node run used '''128'''. When you compare like with like (64 cores on 1 node ≈ 1.11 s versus 64 cores across 4 nodes ≈ 1.09 s) the times are essentially equal: for this workload the network between nodes is not a bottleneck. Always check the core count before reading a comparison as good or bad news.

'''The measurements are deliberately rough.''' Each point is a single run with a single seed, and the fastest runs finish in well under a second — far too short to time precisely on a shared machine. That noise is why a step in the curve can occasionally look ''better'' than perfect ("super-linear"); it is timing jitter, not magic. A proper benchmark repeats each run several times and reports the median. This tutorial trades that rigour for clarity and a short queue.

'''A real-world wrinkle.''' You may notice the single-process MPI run (about 70 s) is markedly slower than the single-thread OpenMP run (about 26 s) for the same work — and that gap persists across the sweep. That is a property of how this particular MPI program starts up and is being investigated; it is not a general rule that MPI is slower than OpenMP. It is a good reminder that ''how'' you measure matters as much as ''what'' you measure.

== Conclusion ==

You ran one calculation on a single core and on hundreds, on CPUs and on GPUs, on one node and across four — and the answer never changed, only the time. That is the core lesson of parallel computing on a cluster: '''you choose a parallel model to match the structure of your problem''', and the reward is speed, not correctness.

A few takeaways to carry forward:

* If your work is made of independent pieces (parameter sweeps, many input files, Monte Carlo trials), it will parallelise well — threads, processes, or a GPU all help.
* If your work has steps that depend on each other, expect communication to limit your speedup, and measure before assuming more cores will help.
* More cores make things faster; only more samples (or a better method) make a Monte Carlo result more accurate.

To go further, try editing a script to use <code>--per-worker</code> instead of <code>--total</code> and watch the difference between weak and strong scaling, or move on to packaging your own software with the [[Tutorials/Apptainer-Introduction|Apptainer tutorials]].

== Troubleshooting ==

* '''<code>myScratch is not set</code>''' — the course environment is not loaded. Load it and try again.
* '''A GPU job sits in the queue (pending)''' — the NVIDIA cards may be busy; the job will start when one frees up. The AMD GPU run uses a reservation and is the reliable one for the course.
* '''You edited a script badly and want a clean copy''' — delete your script folder and re-run the launcher to get fresh copies:
<syntaxhighlight lang="bash">
rm -rf "$myScratch/hpcCourse/pi-tutorial/slurm"
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

== See also ==

* [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] — check your command-line readiness.
* [[Tutorials/Apptainer-Introduction|Apptainer: Introduction]] — packaging software for reproducible runs.
* [[Tutorials|All tutorials]] · [[Workshops]] · [[Training Materials]]

Workflows

Honfi001 — Tue, 28 Jul 2026 19:18:05 GMT

← Older revision		Revision as of 19:18, 28 July 2026
Line 13:		Line 13:

	* [[Workflows/Parallel-Computing\|Parallel Computing]]		* [[Workflows/Parallel-Computing\|Parallel Computing]]
			* [[Workflows/Serial\|Serial]]
	* Embarrassingly Parallel		* Embarrassingly Parallel
	* Multi-threaded		* [[Workflows/Multi-threaded\|Multi-threaded]]
	* Multi-Process Workflows		* Multi-Process Workflows
	* Hybrid Workflows		* Hybrid Workflows

Workflows/Multi-threaded

Honfi001 — Tue, 28 Jul 2026 19:15:59 GMT

Created page with "Serial ran one program on one core. This page takes that same job and lets the program spread its work across '''many''' cores on the '''same''' machine — still one process, but now with several ''threads'' inside it, all sharing the same memory. It is the gentlest real step up the ladder, because for most tools someone else already did the hard part: the program already knows how to use extra cores. Your job is only to give it some, and..."

New page

[[Workflows/Serial-Workflows|Serial]] ran one program on one core. This page takes that same job and lets the program spread its work across '''many''' cores on the '''same''' machine — still one process, but now with several ''threads'' inside it, all sharing the same memory. It is the gentlest real step up the ladder, because for most tools someone else already did the hard part: the program already knows how to use extra cores. Your job is only to give it some, and to tell it how many.

If you have read the serial page you already have the whole script. This one is that script with two lines touched — and seeing exactly which two is the point.

== When this is you ==

You are looking at a multi-threaded workflow when the program itself knows how to use more than one core, and says so:

* Its manual offers a flag like <code>--threads 8</code>, <code>-p 4</code>, or <code>-@ 16</code> — some way to hand it a number of cores.
* Or its documentation mentions '''OpenMP''', and it responds to an <code>OMP_NUM_THREADS</code> setting.
* It runs as a single program — not several cooperating copies, and not across more than one node.

Plenty of everyday tools are built this way: read aligners, <code>samtools</code>, and many of the numerical libraries underneath R and Python. If a tool has a threads option and you have been leaving it at its default of 1, this page is how you turn it up correctly on Anunna. If the program has no such option, it is [[Workflows/Serial-Workflows|serial]] — more cores will not help it, and you should go back one rung.

== What it looks like on Anunna ==

Everything from the serial job still holds: you never run the analysis on the login node, your work lives under <code>$myScratch</code> on Lustre rather than in your home directory, and you describe the job in a script handed to <code>sbatch</code>. Only two lines change.

Start from the serial script, raise the core count, and keep '''one''' task:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=my_analysis
#SBATCH --ntasks=1 # still ONE process ...
#SBATCH --cpus-per-task=16 # ... but now sixteen cores for it
#SBATCH --mem=4G # unchanged: the threads share this memory
#SBATCH --time=01:00:00 # wall-clock limit; the job is killed past this
#SBATCH --output=%x-%j.out # stdout + stderr, named by job name and id

cd "$myScratch/my_analysis"

module load 2024 Python/3.12.3 # replace with your actual module

# Hand the core count Slurm gave you to the program:
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
python myScript.py --input data.csv --output results.csv
</syntaxhighlight>

Against the serial script the difference is exactly two lines. <code>--cpus-per-task</code> went from 1 to 16, and one new line was added just before the program runs. Three things did '''not''' change, and each is deliberate:

* <code>--ntasks</code> stayed at '''1'''. This is one process that grows more arms, not many processes — the distinction that separates this page from [[Workflows/Multi-Process-Workflows|Multi-Process]].
* <code>--mem</code> stayed the same. The threads all read and write the ''same'' memory, so asking for more cores does not multiply your memory need the way separate processes would.
* Everything else — the module, the command, the scratch directory — is untouched.

The one new line depends on how your tool takes its core count. An OpenMP program, like the one above, reads the <code>OMP_NUM_THREADS</code> environment variable. A tool with its own flag takes the same number directly:

<syntaxhighlight lang="bash">
samtools sort --threads ${SLURM_CPUS_PER_TASK} -o sorted.bam input.bam
</syntaxhighlight>

Either way the rule is identical: whatever number you put in <code>--cpus-per-task</code>, feed that same number to the program. Write <code>${SLURM_CPUS_PER_TASK}</code> rather than hard-coding <code>16</code> in two places — then changing the request is a one-line edit and the two can never drift apart.

== The line that does the real work ==

It is tempting to read <code>--cpus-per-task=16</code> as the line that parallelised the job. It is not. That line only ''reserves'' sixteen cores; by itself it does exactly what it did for the serial program — fifteen of them sit idle with your name on them.

The line that actually uses them is the one that passes the count to the program. Slurm sets the cores aside and exports the number as <code>SLURM_CPUS_PER_TASK</code>, but it never tells your software they exist — you do. This is the single most common mistake on this rung: raise <code>--cpus-per-task</code>, forget the hand-off line, and you get a serial run that merely wasted fifteen cores and a longer queue wait to boot.

The [[Tutorials/Parallelism-Estimating-Pi|π tutorial]] makes you do this by hand: its OpenMP job ships with <code>OMP_NUM_THREADS=FIXME</code>, and [[Tutorials/Parallelism-Estimating-Pi#Step 3: Fill in the two blanks|Step 3]] is filling in <code>${SLURM_CPUS_PER_TASK}</code>. If you want the muscle memory for this page, that is where to get it.

== What it buys you ==

The tutorial runs the same ten-billion-dart π estimate at several thread counts on one node. The shape of the result is the whole lesson:

<syntaxhighlight lang="text">
variant cores wall_s
serial 1 26.008
omp 1 26.499
omp 16 1.663
omp 128 0.235
</syntaxhighlight>

One thread costs the serial time, as it should — same work, one core, plus a hair of overhead. Sixteen cores bring it down roughly sixteenfold; 128 cores by over a hundredfold. For this deliberately friendly problem the threads scale almost perfectly.

Real tools rarely scale this cleanly. Some part of every program refuses to be split, and past a point extra threads stop helping — that is [https://en.wikipedia.org/wiki/Amdahl%27s_law Amdahl's law], sketched as intuition on the [[Workflows/Parallel-Computing#Why not just ask for a thousand cores|overview]] and shown with the actual curves in the [[Tutorials/Parallelism-Estimating-Pi#Discussion|tutorial's discussion]]. The practical habit that follows: try a few thread counts on ''your'' data and watch where the speedup flattens, rather than requesting a whole node by reflex.

== The honest limitation ==

Threads share memory, and shared memory means '''one node'''. Every thread must reach the same physical RAM, so a multi-threaded job cannot spill onto a second machine — its hard ceiling is the core count of a single node, and not one core beyond. Anunna's nodes range from tens to hundreds of cores depending on generation (see [[Compute Hardware Overview]]); whichever node you land on, that number is your absolute maximum here.

When a problem needs more than one node's worth of cores, threads alone cannot take you there. That is the boundary where [[Workflows/Multi-Process-Workflows|multi-process (MPI)]] begins — many processes across many nodes — and where [[Workflows/Hybrid-Workflows|hybrid]] workflows combine the two. Do not reach for them early, though: one node of modern cores is a great deal of compute, and staying on a single node keeps your job script simple and your queue wait short. Climb to the next rung only once you have measured yourself hitting this one's ceiling.

== See also ==

* [[Workflows/Parallel-Computing|Parallel Computing]] — the overview, and the map to the other workflow types.
* [[Workflows/Serial-Workflows|Serial Workflows]] — the baseline this page starts from.
* [[Workflows/Multi-Process-Workflows|Multi-Process Workflows]] — the next rung, for when one node is not enough.
* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — its OpenMP rung is the worked example for this page.
* [[Compute Hardware Overview]] — how many cores a single node actually has.

Workflows

Honfi001 — Tue, 28 Jul 2026 19:15:12 GMT

Parallel Workflows

← Older revision		Revision as of 19:15, 28 July 2026
Line 14:		Line 14:
	* [[Workflows/Parallel-Computing\|Parallel Computing]]		* [[Workflows/Parallel-Computing\|Parallel Computing]]
	* Embarrassingly Parallel		* Embarrassingly Parallel
	* Multi-threaded ~~Workflows~~		* Multi-threaded
	* Multi-Process Workflows		* Multi-Process Workflows
	* Hybrid Workflows		* Hybrid Workflows

Storage Requests

Roest039 — Tue, 28 Jul 2026 07:52:21 GMT

Removed the lustre filesystems from the line about increasing what you need, because quota are only set on your home dir

← Older revision		Revision as of 07:52, 28 July 2026
Line 5:		Line 5:
	Submit a request through [[How to Get Help]] (the WUR support portal, '''Anunna (HPC)''' service). It helps to include:		Submit a request through [[How to Get Help]] (the WUR support portal, '''Anunna (HPC)''' service). It helps to include:

	* ~~which filesystem~~ you need ~~more of —~~ home~~, [[Compute Storage\|Lustre]]~~ (~~backup / nobackup / scratch~~)~~, or [[Archival Storage\|/archive]];~~		* how much space you need in your /home directory (normally you get 200GB), and for how long;
	* how much space you need, and for how long;
	* the group or project it is for.		* the group or project it is for.

Support Ticket System

Roest039 — Tue, 28 Jul 2026 06:54:22 GMT

Changed HPC (Anunna) to Anunna (HPC) as it is called on the support page

← Older revision		Revision as of 06:54, 28 July 2026
Line 4:		Line 4:

	# Go to the [https://support.wur.nl/esc WUR support portal].		# Go to the [https://support.wur.nl/esc WUR support portal].
	# Choose to create a ticket and register it on the '''HPC ~~(Anunna~~)''' service.		# Choose to create a ticket and register it on the '''Anunna (HPC)''' service.
	# Describe your issue — see [[Reporting Incidents]] for what to include.		# Describe your issue — see [[Reporting Incidents]] for what to include.

Storage Requests

Roest039 — Tue, 28 Jul 2026 06:51:58 GMT

Changed HPC (Anunna) to Anunna (HPC) as it is called on the support page

← Older revision		Revision as of 06:51, 28 July 2026
Line 3:		Line 3:
	== Requesting more storage ==		== Requesting more storage ==

	Submit a request through [[How to Get Help]] (the WUR support portal, '''HPC ~~(Anunna~~)''' service). It helps to include:		Submit a request through [[How to Get Help]] (the WUR support portal, '''Anunna (HPC)''' service). It helps to include:

	* which filesystem you need more of — home, [[Compute Storage\|Lustre]] (backup / nobackup / scratch), or [[Archival Storage\|/archive]];		* which filesystem you need more of — home, [[Compute Storage\|Lustre]] (backup / nobackup / scratch), or [[Archival Storage\|/archive]];

How to Get Help

Roest039 — Tue, 28 Jul 2026 06:50:24 GMT

WUR users: Changed HPC (Anunna) to Anunna (HPC) as it is called on the support page

← Older revision		Revision as of 06:50, 28 July 2026
Line 5:		Line 5:
	=== WUR users ===		=== WUR users ===

	Submit a support ticket through the [https://support.wur.nl/esc WUR support portal] and select the '''HPC ~~(Anunna~~)''' service. See [[Support Ticket System]] for details.		Submit a support ticket through the [https://support.wur.nl/esc WUR support portal] and select the '''Anunna (HPC)''' service. See [[Support Ticket System]] for details.

	WUR users can also join the '''Anunna Users''' Microsoft Teams team for user-to-user questions and tips — use the join code <code>2xo7wv2</code> ("Join a team with a code"). This is a peer community rather than an official support channel, so for problems that need the HPC team, still file a ticket.		WUR users can also join the '''Anunna Users''' Microsoft Teams team for user-to-user questions and tips — use the join code <code>2xo7wv2</code> ("Join a team with a code"). This is a peer community rather than an official support channel, so for problems that need the HPC team, still file a ticket.

Workflows/Serial-Workflows

Honfi001 — Mon, 27 Jul 2026 10:36:45 GMT

Honfi001 moved page Workflows/Serial-Workflows to Workflows/Serial without leaving a redirect

New page

Every other page in this section describes a way of splitting work across many cores. This one describes ''not'' doing that — one program, running on a single core, from start to finish. It is the baseline, and, to be honest up front, it is not a parallel workflow at all.

It earns a page anyway, for two reasons. Most software you did not ''specifically'' choose for its parallelism runs this way, so this is where many jobs on Anunna may actually belong. And every other workflow in this section is best understood as a small change to the serial job script — so it is worth seeing the starting point clearly before you start changing lines in it.

== When this is you ==

You are almost certainly looking at a serial workflow if any of these are true:

* The program's manual never mentions threads, cores, ranks, MPI, or a GPU.
* It has no flag like <code>--threads</code>, <code>-p</code>, or <code>-n</code> for handing it more cores.
* You run it on your laptop today, and it uses one core there too.

That covers a lot of ground: most command-line bioinformatics tools, most R and Python scripts you did not deliberately parallelise, most one-off analyses. If you are not sure, assume serial — it is the safe default, and climbing the ladder later is easy once the baseline works.

== What it looks like on Anunna ==

You never run the analysis directly on the login node — that machine is shared by everyone and is meant for editing, submitting, and light housekeeping only. Instead you describe the job in a short script and hand it to Slurm with <code>sbatch</code>; Slurm finds a compute node and runs it there.

A minimal serial job script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=my_analysis
#SBATCH --ntasks=1 # one process
#SBATCH --cpus-per-task=1 # one core for that process
#SBATCH --mem=4G # memory, sized to your data
#SBATCH --time=01:00:00 # wall-clock limit; the job is killed past this
#SBATCH --output=%x-%j.out # stdout + stderr, named by job name and id

cd "$myScratch/my_analysis"

module load 2024 Python/3.12.3 # replace with your actual module
python myScript.py --input data.csv --output results.csv
</syntaxhighlight>

Save it as <code>run.slurm</code> and submit:

<syntaxhighlight lang="bash">
sbatch run.slurm
</syntaxhighlight>

Two things to notice, because the other pages in this section keep coming back to them:

* Work lives under your lustre directories, e.g. <code>$myScratch</code>, not in your home directory. Lustre is the fast, roomy filesystem built for job data; home is small and is not meant to handle computations.
* The two lines that decide how much compute the job gets are '''<code>--ntasks</code>''' and '''<code>--cpus-per-task</code>''', and here they are both '''1'''. Every other workflow in this section is essentially this same script with one of those two numbers raised — plus a matching change telling the program to actually use what it was given.

== Why asking for more cores does nothing ==

It is tempting to bump <code>--cpus-per-task</code> to 8 and hope for an eightfold speedup. For a serial program it buys you nothing.

A serial program is a single stream of instructions with nowhere to put a second core. Slurm will dutifully reserve all eight if you ask, but seven of them sit idle '''with your name on them''' for the entire run. You also wait longer in the queue for eight free cores than for one, and if the cluster does fairshare accounting you may be charged for the idle seven regardless. The job does not finish a moment sooner. The only thing that makes a serial program use more than one core is '''rewriting it''' to be parallel — which is what the rest of this section is about.

The one resource worth sizing on a serial job is <code>--mem</code>: give it enough memory for your data, and no more.

== The honest limitation ==

Serial is not a parallel workflow, and no Slurm flag changes that. Its ceiling is one core's worth of speed, full stop.

That is not a problem until it is — until the run takes four days, or the dataset no longer fits in memory. When you hit that wall, the script above is your launch pad, and the only question is ''which'' line to change:

* Same program, many independent inputs — a parameter sweep, two hundred samples? You want [[Workflows/Embarrassingly-Parallel|Embarrassingly Parallel]]: the job array, and often no change to the program at all.
* The program itself offers a <code>--threads</code> option? That is [[Workflows/Multi-Threaded-Workflows|Multi-Threaded]] — raise <code>--cpus-per-task</code> and point the tool at it.
* The program is built on MPI? That is [[Workflows/Multi-Process-Workflows|Multi-Process]] — raise <code>--ntasks</code> and launch with <code>mpirun</code>.

If you would rather see all of these run side by side on one small problem — with this serial job as the reference everything else is measured against — work through [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]]. Its serial rung plays exactly this role, and the [[Tutorials/Parallelism-Estimating-Pi#Discussion|discussion]] shows what climbing the ladder does and does not buy you.

== See also ==

* [[Workflows/Parallel-Computing|Parallel Computing]] — the overview, and the map to the other workflow types.
* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — the same job run serially and every other way.
* [[Compute Hardware Overview]] — what a single core is actually attached to.
* [[Workflows|Workflows]] — job submission, debugging, and checkpointing.

Workflows/Serial-Workflows

Honfi001 — Mon, 27 Jul 2026 10:35:46 GMT

Created page with "Every other page in this section describes a way of splitting work across many cores. This one describes ''not'' doing that — one program, running on a single core, from start to finish. It is the baseline, and, to be honest up front, it is not a parallel workflow at all. It earns a page anyway, for two reasons. Most software you did not ''specifically'' choose for its parallelism runs this way, so this is where many jobs on Anunna may actually belong. And every othe..."

New page

Workflows/Parallel-Computing

Honfi001 — Thu, 23 Jul 2026 15:40:28 GMT

Why not just ask for a thousand cores

← Older revision		Revision as of 15:40, 23 July 2026
(One intermediate revision by the same user not shown)
Line 27:		Line 27:
	Because you would not get a thousandfold speedup, and usually not anything close to it.		Because you would not get a thousandfold speedup, and usually not anything close to it.

	Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''Amdahl's law''', and the practical version of it is: '''measure before you ask for more'''.		Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''[https://en.wikipedia.org/wiki/Amdahl%27s_law Amdahl's law]''', and the practical version of it is: '''measure before you ask for more'''.

	Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts ~~whithin~~ ten minutes often finishes long before the greedy one has even begun.		Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts within ten minutes often finishes long before the greedy one has even begun.

	If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion\|π tutorial's discussion]] has the timings and the scaling curves.		If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion\|π tutorial's discussion]] has the timings and the scaling curves.

Workflows

Honfi001 — Thu, 23 Jul 2026 15:23:31 GMT

Parallel Workflows

← Older revision		Revision as of 15:23, 23 July 2026
Line 12:		Line 12:
	=== Parallel Workflows ===		=== Parallel Workflows ===

	* Parallel Computing		* [[Workflows/Parallel-Computing\|Parallel Computing]]
	* Embarrassingly Parallel		* Embarrassingly Parallel
	* Multi-threaded Workflows		* Multi-threaded Workflows

Workflows/Paralle-Computing

Honfi001 — Thu, 23 Jul 2026 15:22:44 GMT

Honfi001 moved page Workflows/Paralle-Computing to Workflows/Parallel-Computing without leaving a redirect

← Older revision		Revision as of 15:22, 23 July 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	Your analysis runs. It just runs for too long, or it no longer fits on one machine. '''Parallel computing''' is the answer to both, and this page is the map: what it actually means on Anunna, and which of the six workflow types below is yours.		Your analysis runs. It just runs for too long, or it no longer fits on one machine. '''Parallel computing''' may be the answer to both, and this page is the map: what it actually means on Anunna, and which of the six workflow types below is yours.

	Each section is a short paragraph and a link. Read them, recognise your own problem in one of them, and follow it. If you would rather learn by doing, the [[Tutorials/Parallelism-Estimating-Pi\|π estimation tutorial]] walks the same ladder with real code and real timings.		Each section is a short paragraph and a link. Read them, recognise your own problem in one of them, and follow it. If you would rather learn by doing, the [[Tutorials/Parallelism-Estimating-Pi\|π estimation tutorial]] walks the same ladder with real code and real timings.
Line 29:		Line 29:
	Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''Amdahl's law''', and the practical version of it is: '''measure before you ask for more'''.		Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''Amdahl's law''', and the practical version of it is: '''measure before you ask for more'''.

	Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts in ten minutes often finishes long before the greedy one has even begun.		Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts whithin ten minutes often finishes long before the greedy one has even begun.

	If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion\|π tutorial's discussion]] has the timings and the scaling curves.		If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion\|π tutorial's discussion]] has the timings and the scaling curves.

Workflows/Paralle-Computing

Honfi001 — Thu, 23 Jul 2026 15:18:37 GMT

Created page with "Your analysis runs. It just runs for too long, or it no longer fits on one machine. '''Parallel computing''' is the answer to both, and this page is the map: what it actually means on Anunna, and which of the six workflow types below is yours. Each section is a short paragraph and a link. Read them, recognise your own problem in one of them, and follow it. If you would rather learn by doing, the π estimation tutorial walks the sa..."

New page

Your analysis runs. It just runs for too long, or it no longer fits on one machine. '''Parallel computing''' is the answer to both, and this page is the map: what it actually means on Anunna, and which of the six workflow types below is yours.

Each section is a short paragraph and a link. Read them, recognise your own problem in one of them, and follow it. If you would rather learn by doing, the [[Tutorials/Parallelism-Estimating-Pi|π estimation tutorial]] walks the same ladder with real code and real timings.

== Why bother ==

Nothing glamorous. The reasons are almost always one of these:

* '''The run takes too long.''' Four days for one analysis means four days before you find out whether the parameters were right.
* '''The dataset outgrew the machine.''' What fitted comfortably on a laptop last year now needs more memory than a laptop has.
* '''There are simply a lot of runs.''' Two hundred samples, one pipeline, one after another after another.
* '''The hardware is sitting idle.''' Anunna's compute nodes carry anywhere from tens to hundreds of cores each, depending on which generation you land on (see [[Compute Hardware Overview]]). A serial job uses exactly one of them and leaves the rest with nothing to do.

A cluster does not make your program faster. It lets '''many pieces of the work happen at once''' — but only if the work can be divided, and only if your program has been told to divide it. Slurm hands you the cores; it does not teach your software to use them.

== The one idea ==

Underneath all the jargon there are only two ways to divide work:

* '''Split the data.''' Every worker gets a different slice of the same job — different samples, different rows, different darts.
* '''Split the steps.''' Every worker takes a different part of the calculation, and they exchange results as they go.

Everything below is a variation on one of those two. What differs is ''who the workers are'' — threads, processes, GPU device threads — and ''how much they have to talk to each other''. That second part is what separates the easy cases from the hard ones, so it is worth noticing which side of it your problem sits on.

== Why not just ask for a thousand cores ==

Because you would not get a thousandfold speedup, and usually not anything close to it.

Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''Amdahl's law''', and the practical version of it is: '''measure before you ask for more'''.

Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts in ten minutes often finishes long before the greedy one has even begun.

If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion|π tutorial's discussion]] has the timings and the scaling curves.

== Which workflow is yours ==

Six types, roughly in order of how much you have to think about. You should be able to recognise yourself in one of them.

'''Serial — one process, one core, start to finish.''' This is the baseline, and it is honestly not a parallel workflow at all. It is here because every other type is best described as a change to it: same job script, one line different. Most software you did not specifically choose for its parallelism is serial, and asking Slurm for more cores will not change that — the extra cores sit idle with your name on them. Start here so the other five have something to differ from. Full page: [[Workflows/Serial-Workflows|Serial Workflows]].

'''Embarrassingly parallel — the same program, many times, on different inputs.''' Two hundred samples through one pipeline. A parameter sweep. A hundred simulations that differ only by their random seed. The runs never need to talk to each other, which makes this both the easiest kind of parallelism and by far the most common in biology. Slurm has a purpose-built tool for it, the job array, and it usually needs no change to your program whatsoever. If your problem fits here, you are in luck. Full page: [[Workflows/Embarrassingly-Parallel|Embarrassingly Parallel]].

'''Multi-threaded — one program, many cores, shared memory.''' The program stays a single process but splits its work across several threads, all reading and writing the same data in memory. This is what a tool means when its manual offers <code>--threads 16</code> or <code>-p 8</code>. It is the gentlest real parallelism to adopt, because someone else already did the hard part. Its hard limit is that every thread must live on the same node: you can grow to that node's core count and not one core further. Full page: [[Workflows/Multi-Threaded-Workflows|Multi-Threaded Workflows]].

'''Multi-process — many processes that talk to each other.''' When one node is not enough, the work is split across separate processes, each with its own private memory, passing data over the network as the calculation proceeds. The standard for this is MPI, and it is what lets a single job span the cluster. The catch is that the program must have been written for MPI from the inside; it is not something you can bolt on from the job script. Full page: [[Workflows/Multi-Process-Workflows|Multi-Process Workflows]].

'''Hybrid — processes between nodes, threads within them.''' A few MPI processes per node, each one running a handful of threads. This buys the reach of MPI with less of its communication overhead, and it is how most large simulation codes actually run at scale. The price is two tuning knobs that interact, and arithmetic that has to come out right — processes times threads must match what you asked Slurm for. This is the one people most often get subtly wrong. Full page: [[Workflows/Hybrid-Workflows|Hybrid Workflows]].

'''GPU — hand the heavy arithmetic to an accelerator.''' A GPU runs thousands of very simple workers at once and is spectacular on the narrow class of problems shaped to suit it. Anunna has both NVIDIA and AMD cards, in separate partitions — see [[Compute Hardware Overview]] for what is actually installed. The limitation here is absolute rather than a matter of degree: the software must already have a GPU version. Most software does not, and no Slurm flag will conjure one. Full page: [[Workflows/GPU-Workflows|GPU Workflows]].

== Where to go next ==

Once you know which type is yours, its page shows what the job script looks like and which modules go with it.

If you would rather see all of them side by side on one problem first, work through [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]]. It takes a single calculation — throwing random darts at a dartboard — and runs it serially, with threads, with processes, across several nodes, and on a GPU. The answer never changes. Only the time does, which is exactly the point.

== See also ==

* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — the hands-on companion to this page.
* [[Compute Hardware Overview]] — what the nodes and GPUs actually are.
* [[Workflows|Workflows]] — job submission, debugging, and checkpointing.
* [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] — check your command-line readiness first.

Network & Security

Honfi001 — Thu, 23 Jul 2026 15:07:49 GMT

← Older revision		Revision as of 15:07, 23 July 2026
Line 5:		Line 5:
	== Network ==		== Network ==

	Internally inside the cluster, the compute nodes and the[[Filesystems \| Lustre filesystem]] are linked by a high-speed, low-latency interconnect, which is what lets the parallel filesystem and multi-node jobs perform well.		Internally inside the cluster, the compute nodes and the[[Filesystems \| Lustre filesystem]] are linked by OmniPath, a high-speed, low-latency interconnect, which is what lets the parallel filesystem and multi-node jobs perform well.

	From outside, the cluster services are reached through two loadbalancers, which are connected to the internet over 100Gbit links.		From outside, the cluster services are reached through two loadbalancers, which are connected to the internet over 100Gbit links.
Line 13:		Line 13:
	== Authentication ==		== Authentication ==

	~~You log in with your~~ WUR ~~or outside group~~ account~~. For stronger protection you can add multi-factor authentication~~ to your SSH access using a hardware security key — see [[Using a hardware key for better security]]. You can also configure key-based login so you are not prompted for a password each time; see [[Ssh without password]].		Anunna is public facing, servicing organisation withing and outside of the WUR but access requires an account to be activated by the HPC admins.

	If you use the wrong credentials, your access ~~will~~ be blocked, see [[Unblocking your IP]]		Users from the WUR can request an account and once the account is activated, they can login with the same credentials as their WUR account.

			In a similar fashion, organisations outside of the WUR can request submit account requests to the admins. Once the account is created, the users receive an email with their login details, which they are advised to change when first logging in.

			For stronger protection you can add multi-factor authentication to your SSH access using a hardware security key — see [[Using a hardware key for better security]]. You can also configure key-based login so you are not prompted for a password each time; see [[Ssh without password]].

			If you use the wrong credentials, your access may be blocked, see [[Unblocking your IP]]

	== Data security ==		== Data security ==

System Status Page

Prins089 — Thu, 23 Jul 2026 12:49:43 GMT

Prins089 deleted page System Status Page The wiki is not meant for status tracking, it's a knowledge base. We need an actual status page (a pod in kubernetes with rudimentary monitoring), and when we have that, let's link to it from the wiki. But now we do not have this

Workflows

Honfi001 — Thu, 23 Jul 2026 12:25:11 GMT

← Older revision		Revision as of 12:25, 23 July 2026
(2 intermediate revisions by the same user not shown)
Line 12:		Line 12:
	=== Parallel Workflows ===		=== Parallel Workflows ===

	* [[Performance Optimization/Multiple CPUs]] — using several cores in one job		* Parallel Computing
	* [[Performance Optimization/Multiple nodes (MPI)]] — scaling across nodes with MPI		* Embarrassingly Parallel
	* [[Performance Optimization/Multiple nodes (arrayjobs)]] — scaling with job arrays		* Multi-threaded Workflows
			* Multi-Process Workflows
			* Hybrid Workflows
			* GPU workflows

			* <s>[[Performance Optimization/Multiple CPUs]] — using several cores in one job</s>
			* <s>[[Performance Optimization/Multiple nodes (MPI)]] — scaling across nodes with MPI</s>
			* <s>[[Performance Optimization/Multiple nodes (arrayjobs)]] — scaling with job arrays</s>

Conda for teaching

Honfi001 — Thu, 23 Jul 2026 11:48:13 GMT

Honfi001 deleted page Conda for teaching content was: "{{Warning|This page describes the older cluster setup: the <code>/cm/shared/...</code> paths and the tcl modulefiles below are from the previous (Bright) cluster and do '''not''' match current Anunna, which uses the Lustre filesystem and EasyBuild buckets. The general approach — a shared conda environment registered as a Jupyter kernel — still applies, but the paths and commands need updating. For the current way to set up course kernels, see Steps for courses and Pyt..."

Workflow Engines (Snakemake, Nextflow)

Honfi001 — Wed, 22 Jul 2026 18:55:35 GMT

Pitfalls

Show changes

Workflows

Honfi001 — Tue, 21 Jul 2026 14:15:49 GMT

Replace performance optimizations with Parallel Workflows

← Older revision		Revision as of 14:15, 21 July 2026
Line 10:		Line 10:
	* [[Scheduled tasks (cron)]] — running recurring tasks with scrontab		* [[Scheduled tasks (cron)]] — running recurring tasks with scrontab

	=== ~~Performance optimization~~ ===		=== Parallel Workflows ===

	* [[Performance Optimization/Multiple CPUs]] — using several cores in one job		* [[Performance Optimization/Multiple CPUs]] — using several cores in one job
	* [[Performance Optimization/Multiple nodes (MPI)]] — scaling across nodes with MPI		* [[Performance Optimization/Multiple nodes (MPI)]] — scaling across nodes with MPI
	* [[Performance Optimization/Multiple nodes (arrayjobs)]] — scaling with job arrays		* [[Performance Optimization/Multiple nodes (arrayjobs)]] — scaling with job arrays

Reporting Usage

Prins089 — Tue, 21 Jul 2026 12:30:48 GMT

← Older revision		Revision as of 12:30, 21 July 2026
Line 3:		Line 3:
	== Estimating your costs ==		== Estimating your costs ==

	The <code>get_my_bill</code> script estimates ~~your personal~~ costs over a period. Load the <code>anunna</code> module and run it:		The <code>get_my_bill</code> script shows your Slurm jobs and estimates their compute costs over a period. Load the <code>anunna</code> module and run it:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 10:		Line 10:
	</syntaxhighlight>		</syntaxhighlight>

	By default it ~~reports~~ your ~~cost for~~ the current month ~~so far~~, ~~broken down by account (project)~~ and ~~by storage tier~~. The ~~figures~~ are ~~advisory~~.		By default, it shows jobs for your user account from the first day of the current month through the current time. Each row includes the job start time, job ID and name, state, allocated CPUs, GPUs and memory, elapsed seconds, and estimated cost. A total is shown at the bottom.

			The estimate is based on the resources allocated to each job and its elapsed wall time. It is indicative rather than a final bill; the monthly usage reports are leading.

	Useful options:		Useful options:

	* <code>-g</code> — ~~show results for your entire group~~.		* <code>-s YYYY-MM-DD</code> or <code>--start YYYY-MM-DD</code> — start the report on this date. The default is the first day of the current month.
	* <code>-~~d compute~~</code> or <code>-~~d disk~~</code> — ~~extra detail~~ on ~~compute~~ or ~~disk~~ usage.		* <code>-e YYYY-MM-DD</code> or <code>--end YYYY-MM-DD</code> — end the report on this date, inclusive. The default is the current time.
	* <code>-~~t YYYY~~-MM</code> — ~~results~~ for a ~~specific month.~~		* <code>-u USER</code> or <code>--user USER</code> — report usage for another Slurm user.
			* <code>-g</code> or <code>--group</code> — report all jobs charged to the selected user's default Slurm account. Group output includes a <code>User</code> column.
			* <code>-h</code> or <code>--help</code> — show all available options.

			Long options that take a value can use either a space or an equals sign, for example <code>--start 2026-01-01</code> and <code>--start=2026-01-01</code> are equivalent.

			Examples:

			<syntaxhighlight lang="bash">
			# Your jobs from a given date through now
			get_my_bill --start 2026-01-01

			# Your jobs in an inclusive date range
			get_my_bill -s 2026-01-01 -e 2026-01-31

			# All jobs charged to your group
			get_my_bill --group
			</syntaxhighlight>

	== Cost alerts ==		== Cost alerts ==

Batch Jobs

Honfi001 — Tue, 21 Jul 2026 12:10:36 GMT

Loading modules

← Older revision		Revision as of 12:10, 21 July 2026
(One intermediate revision by the same user not shown)
Line 22:		Line 22:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module ~~avail~~ Python		module spider Python
	</syntaxhighlight>		</syntaxhighlight>

			Note that the case of the module matters. This command will give you a list of all Python modules available in the cluster.

	Pick one and load it from the appropriate year bucket — see [[Modules]] for the bucket-and-version convention.		Pick one and load it from the appropriate year bucket — see [[Modules]] for the bucket-and-version convention.
Line 33:		Line 35:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	#SBATCH --comment=~~773320000~~		#SBATCH --comment=01189998819991197253
	#SBATCH --time=1200		#SBATCH --time=1200
	#SBATCH --mem=2048		#SBATCH --mem=2048
Line 173:		Line 175:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	#SBATCH --comment=~~773320000~~		#SBATCH --comment=01189998819991197253
	</syntaxhighlight>		</syntaxhighlight>

Performance Optimization/Multiple nodes (MPI)

Honfi001 — Tue, 21 Jul 2026 11:34:14 GMT

Running an MPI program

← Older revision		Revision as of 11:34, 21 July 2026
Line 7:		Line 7:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	module purge		module purge
	module load ~~2024~~		module load 2025
	module load ~~gcc openmpi~~/~~gcc slurm~~		module load OpenMPI/5.0.7
	</syntaxhighlight>		</syntaxhighlight>

Line 53:		Line 53:

	module purge		module purge
	module load ~~2024~~		module load 2025
	module load ~~gcc openmpi~~/~~gcc slurm~~		module load OpenMPI/5.0.7

	srun ./hello_mpi		srun ./hello_mpi
	</syntaxhighlight>		</syntaxhighlight>

Compute Hardware Overview

Honfi001 — Tue, 21 Jul 2026 07:40:33 GMT

← Older revision		Revision as of 07:40, 21 July 2026
Line 88:		Line 88:
	!GPU Maker		!GPU Maker
	!GPU Type		!GPU Type
	!~~Number of~~ GPUs		!GPUs per Node
	!VRAM (GB)		!VRAM (GB)
	!VRAM Type		!VRAM Type