View source for Workflows/Embarassinly-Parallel

This tutorial is a gentle, hands-on introduction to '''parallel computing on Anunna'''. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the ''same'' calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you.

You do not need to write any C code. The whole exercise comes down to filling in '''two''' blanks in two job scripts, launching a batch of jobs, and reading three plots.

== What you will learn ==

* How a single program can be run serially, with threads (OpenMP), with processes (MPI), across several nodes, and on a GPU.
* How Slurm ''allocates'' resources and why your program still has to be ''told'' how many to use.
* The difference between '''strong scaling''' (same work, more workers, less time) and '''weak scaling''' (more workers, more work, same time).
* Why more workers make a calculation '''faster, not more accurate'''.
* How to read scaling and speedup plots — and where their limits are.

== Before you begin ==

You should already be comfortable logging into Anunna and submitting a basic Slurm job. If you are not sure, work through the [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] first.

You also need the '''course environment loaded''', because it sets the <code>$myScratch</code> variable that the tutorial relies on. If your instructor has not told you otherwise, it is loaded for you when you log in. You can check with:

<syntaxhighlight lang="bash">
echo "$myScratch"
</syntaxhighlight>

If that prints a path under <code>/lustre/scratch/...</code>, you are ready. If it prints nothing, load the course environment before continuing.

Everything you run lives in two places:

* '''The shared tutorial''' at <code>/lustre/shared/hpcCourses/pi-tutorial</code> — read-only, the same for everyone.
* '''Your personal copy''' at <code>$myScratch/hpcCourse/pi-tutorial</code> — created the first time you launch, and where you make your edits.

== The big idea: estimating π with random darts ==

Imagine a square dartboard with a quarter-circle drawn inside it. If you throw darts at random and they land anywhere in the square with equal chance, the fraction that land ''inside'' the quarter-circle is exactly the ratio of the two areas — which works out to π/4. So if you throw '''N''' darts and '''H''' of them land inside:

<syntaxhighlight lang="text">
π ≈ 4 × H / N
</syntaxhighlight>

Throw more darts, and the estimate gets better. This is a '''Monte Carlo''' method: a calculation built out of huge numbers of independent random trials. It has one property that makes it perfect for teaching parallelism — every dart is independent, so the work splits cleanly across as many workers as you like, with almost no coordination between them.

The tutorial implements this ''same'' dart-throwing kernel six ways:

{| class="wikitable"
! Rung !! Program !! What it uses
|-
| serial || <code>pi_serial</code> || one core (the reference)
|-
| OpenMP || <code>pi_omp</code> || many threads on one node
|-
| MPI || <code>pi_mpi</code> || many processes on one node
|-
| multi-node || <code>pi_mpi</code> || processes spread across several nodes (same program)
|-
| GPU || <code>pi_gpu</code> || thousands of device threads on one GPU
|-
| hybrid || <code>pi_hybrid</code> || MPI processes, each running OpenMP threads
|}

Because the random-number generator is identical everywhere and each worker draws its own independent stream, the ''answer'' for a given number of darts and seed is the same no matter how the work is divided. Only the ''time'' changes. That is the whole point.

== Step 1: Make the tutorial modules available ==

The tutorial ships as a set of environment modules. Point Lmod at them once per session:

<syntaxhighlight lang="bash">
module use /lustre/shared/hpcCourses/pi-tutorial/modules
module avail pi-tutorial
</syntaxhighlight>

You will see three modules. They are mutually exclusive — loading one unloads the others:

* <code>pi-tutorial</code> — the CPU rungs (serial, OpenMP, MPI, hybrid).
* <code>pi-tutorial-cuda</code> — the NVIDIA GPU rung.
* <code>pi-tutorial-rocm</code> — the AMD GPU rung.

You do not need to load these by hand for the exercise — each job script loads the right one for you. The <code>module use</code> line above is enough.

== Step 2: Get your personal copy of the scripts ==

Run the launcher once:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

On this first run it does three things: it copies the job scripts into your personal space at <code>$myScratch/hpcCourse/pi-tutorial/slurm</code>, clears any old results, and submits the whole batch of jobs. Watch them with:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Most of the jobs (serial, GPU, multi-node, hybrid) are complete and will run correctly. '''Two of them — the OpenMP and the MPI jobs — contain a deliberate blank''' and will not give a useful result yet. Fixing those blanks is your job, and it is the next step.

== Step 3: Fill in the two blanks ==

When you ask Slurm for resources, Slurm sets aside the cores or tasks you requested — but it does not automatically tell your ''program'' about them. You have to pass the number along. Slurm makes that easy by exporting the counts as environment variables. Your task is to connect the two.

Open the OpenMP script in your personal copy (use <code>nano</code> if you are not comfortable with <code>vi</code>):

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multithread.slurm
</syntaxhighlight>

You will find this line:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=FIXME
</syntaxhighlight>

The script asked Slurm for cores with <code>--cpus-per-task</code>, and Slurm exported that count as <code>SLURM_CPUS_PER_TASK</code>. Tell OpenMP to use exactly that many threads:

<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
</syntaxhighlight>

Now the MPI script:

<syntaxhighlight lang="bash">
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multiprocess.slurm
</syntaxhighlight>

Here the blank is the number of processes to launch:

<syntaxhighlight lang="bash">
mpirun -np FIXME pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

This script asked Slurm for tasks with <code>--ntasks</code>, and Slurm exported that as <code>SLURM_NTASKS</code>. Launch one process per task:

<syntaxhighlight lang="bash">
mpirun -np ${SLURM_NTASKS} pi_mpi --total 1e10 --seed 12345
</syntaxhighlight>

That is the entire editable surface — two lines. Save both files.

== Step 4: Launch the full sweep ==

Run the launcher again:

<syntaxhighlight lang="bash">
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

This time it '''keeps your edited scripts''' (it never overwrites your copy), clears the old results, and resubmits everything. The launcher sweeps a range of sizes for you automatically — OpenMP from 1 up to 128 threads, MPI from 1 up to 128 processes — plus the GPU, multi-node, and hybrid runs.

Keep an eye on the queue:

<syntaxhighlight lang="bash">
squeue --me
</syntaxhighlight>

Each finished job writes one result line into <code>$myScratch/hpcCourse/pi-tutorial/results</code>. Wait until the queue is empty before moving on.

== Step 5: Analyze your results ==

The analysis script reads every result file, prints a summary table, and writes three plots. It needs matplotlib, which lives in the SciPy bundle:

<syntaxhighlight lang="bash">
module load 2024 SciPy-bundle/2024a
python /lustre/shared/hpcCourses/pi-tutorial/analyze.py $myScratch/hpcCourse/pi-tutorial/results
</syntaxhighlight>

When it finishes you will have three PNG files in your results directory:

* <code>walltime_vs_cores.png</code> — how long each run took, against the number of cores.
* <code>speedup_vs_cores.png</code> — how much faster than the serial run, against the ideal straight line.
* <code>best_per_approach.png</code> — the single fastest run of each approach, side by side.

Copy them to your laptop (for example with <code>scp</code>) to view them, or open them through the OnDemand file browser.

== Results ==

[[File:Pi-tutorial-walltime-vs-cores.png|thumb|right|360px|Wall time falls as cores are added. Note both axes are logarithmic; a straight line here means near-ideal scaling.]]
[[File:Pi-tutorial-speedup-vs-cores.png|thumb|right|360px|Speedup against the serial baseline. The dashed line is perfect (linear) speedup; real runs sit just below it.]]
[[File:Pi-tutorial-best-per-approach.png|thumb|right|360px|The fastest run of each approach. The GPU finishes the same 10-billion-dart job in a fraction of the time the CPU rungs need.]]

A correct run produces a table like the one below (trimmed; your exact times will vary by a few percent). Every run threw the same '''10 billion''' darts.

<syntaxhighlight lang="text">
variant   nodes cores              device       wall_s            pi   pct_err
serial        1     1                 cpu       26.008     3.1416052   0.00040
omp           1     1                 cpu       26.499     3.1416052   0.00040
omp           1    16                 cpu        1.663     3.1415960   0.00011
omp           1   128                 cpu        0.235     3.1416075   0.00047
mpi           1     1                 cpu       69.676     3.1416052   0.00040
mpi           1   128                 cpu        0.555     3.1416075   0.00047
mpi           4    64                 cpu        1.090     3.1416039   0.00036
hybrid        4   128                 cpu        0.217     3.1416075   0.00047
gpu           1     1  AMD_Instinct_MI210        0.080     3.1415897   0.00009
gpu           1     1  NVIDIA_A100_80GB_PCIe     0.045     3.1415897   0.00009
</syntaxhighlight>

Two things are worth noticing straight away:

* The serial run took about '''26 seconds'''; the 128-thread OpenMP run took about '''0.24 seconds'''; the A100 GPU finished in about '''0.045 seconds'''. The work was identical — only the time changed.
* Look at the <code>pi</code> column. For any given total work and seed, the estimate is '''identical''' across serial, OpenMP, MPI, multi-node, and hybrid — for example every 128-core CPU run lands on 3.1416075. Splitting the darts across more workers did not change the answer, because each worker draws its own independent random stream. (The GPU differs in the last few digits because it uses far more streams; it still lands well within the expected error.)

== Discussion ==

The plots tell a tidy story: add workers, get proportionally faster, almost along the ideal line. That is real — but it is important to understand ''why'' it looks so clean, and where the limits are.

'''This problem is unusually friendly.''' Throwing darts is "embarrassingly parallel": the workers never need to talk to each other until the very end, when their hit-counts are summed. Most real scientific codes are not like this. A fluid-dynamics solver, a molecular-dynamics simulation, or a genome assembler has data dependencies and communication between workers, and that communication eventually caps how much faster you can go (this ceiling is known as '''Amdahl's law'''). Do not expect your own code to scale as cleanly as π does — this tutorial shows you the ''best'' case, not the typical one.

'''More workers buy speed, not accuracy.''' The error of a Monte Carlo estimate shrinks with the number of darts as roughly 1/√N — so to halve the error you must throw '''four times''' as many darts. Adding cores lets you throw a fixed number of darts ''sooner'', or throw ''more'' darts in the same time, but it never makes a fixed number of darts more accurate. Speed and accuracy are separate knobs.

'''Compare equal core counts.''' In the table, the 4-node MPI run (1.09 s) looks ''slower'' than the 1-node MPI run (0.56 s). It is not a penalty for crossing nodes — the 4-node run used '''64''' cores while the 1-node run used '''128'''. When you compare like with like (64 cores on 1 node ≈ 1.11 s versus 64 cores across 4 nodes ≈ 1.09 s) the times are essentially equal: for this workload the network between nodes is not a bottleneck. Always check the core count before reading a comparison as good or bad news.

'''The measurements are deliberately rough.''' Each point is a single run with a single seed, and the fastest runs finish in well under a second — far too short to time precisely on a shared machine. That noise is why a step in the curve can occasionally look ''better'' than perfect ("super-linear"); it is timing jitter, not magic. A proper benchmark repeats each run several times and reports the median. This tutorial trades that rigour for clarity and a short queue.

'''A real-world wrinkle.''' You may notice the single-process MPI run (about 70 s) is markedly slower than the single-thread OpenMP run (about 26 s) for the same work — and that gap persists across the sweep. That is a property of how this particular MPI program starts up and is being investigated; it is not a general rule that MPI is slower than OpenMP. It is a good reminder that ''how'' you measure matters as much as ''what'' you measure.

== Conclusion ==

You ran one calculation on a single core and on hundreds, on CPUs and on GPUs, on one node and across four — and the answer never changed, only the time. That is the core lesson of parallel computing on a cluster: '''you choose a parallel model to match the structure of your problem''', and the reward is speed, not correctness.

A few takeaways to carry forward:

* If your work is made of independent pieces (parameter sweeps, many input files, Monte Carlo trials), it will parallelise well — threads, processes, or a GPU all help.
* If your work has steps that depend on each other, expect communication to limit your speedup, and measure before assuming more cores will help.
* More cores make things faster; only more samples (or a better method) make a Monte Carlo result more accurate.

To go further, try editing a script to use <code>--per-worker</code> instead of <code>--total</code> and watch the difference between weak and strong scaling, or move on to packaging your own software with the [[Tutorials/Apptainer-Introduction|Apptainer tutorials]].

== Troubleshooting ==

* '''<code>myScratch is not set</code>''' — the course environment is not loaded. Load it and try again.
* '''A GPU job sits in the queue (pending)''' — the NVIDIA cards may be busy; the job will start when one frees up. The AMD GPU run uses a reservation and is the reliable one for the course.
* '''You edited a script badly and want a clean copy''' — delete your script folder and re-run the launcher to get fresh copies:
<syntaxhighlight lang="bash">
rm -rf "$myScratch/hpcCourse/pi-tutorial/slurm"
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
</syntaxhighlight>

== See also ==

* [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] — check your command-line readiness.
* [[Tutorials/Apptainer-Introduction|Apptainer: Introduction]] — packaging software for reproducible runs.
* [[Tutorials|All tutorials]] · [[Workshops]] · [[Training Materials]]