Tutorials/Parallelism-Estimating-Pi
This tutorial is a gentle, hands-on introduction to parallel computing on Anunna. Instead of a heavy scientific code, it uses one friendly problem — estimating the number π by throwing random darts — and runs the same calculation across every kind of parallelism the cluster offers: a single core, many threads, many processes, multiple nodes, and a GPU. Because the problem stays the same, you can see exactly what each step up the ladder buys you.
You do not need to write any C code. The whole exercise comes down to filling in two blanks in two job scripts, launching a batch of jobs, and reading three plots.
What you will learn
- How a single program can be run serially, with threads (OpenMP), with processes (MPI), across several nodes, and on a GPU.
- How Slurm allocates resources and why your program still has to be told how many to use.
- The difference between strong scaling (same work, more workers, less time) and weak scaling (more workers, more work, same time).
- Why more workers make a calculation faster, not more accurate.
- How to read scaling and speedup plots — and where their limits are.
Before you begin
You should already be comfortable logging into Anunna and submitting a basic Slurm job. If you are not sure, work through the Linux Self Assessment first.
You also need the course environment loaded, because it sets the $myScratch variable that the tutorial relies on. If your instructor has not told you otherwise, it is loaded for you when you log in. You can check with:
echo "$myScratch"
If that prints a path under /lustre/scratch/..., you are ready. If it prints nothing, load the course environment before continuing.
Everything you run lives in two places:
- The shared tutorial at
/lustre/shared/hpcCourses/pi-tutorial— read-only, the same for everyone. - Your personal copy at
$myScratch/hpcCourse/pi-tutorial— created the first time you launch, and where you make your edits.
The big idea: estimating π with random darts
Imagine a square dartboard with a quarter-circle drawn inside it. If you throw darts at random and they land anywhere in the square with equal chance, the fraction that land inside the quarter-circle is exactly the ratio of the two areas — which works out to π/4. So if you throw N darts and H of them land inside:
π ≈ 4 × H / N
Throw more darts, and the estimate gets better. This is a Monte Carlo method: a calculation built out of huge numbers of independent random trials. It has one property that makes it perfect for teaching parallelism — every dart is independent, so the work splits cleanly across as many workers as you like, with almost no coordination between them.
The tutorial implements this same dart-throwing kernel six ways:
| Rung | Program | What it uses |
|---|---|---|
| serial | pi_serial |
one core (the reference) |
| OpenMP | pi_omp |
many threads on one node |
| MPI | pi_mpi |
many processes on one node |
| multi-node | pi_mpi |
processes spread across several nodes (same program) |
| GPU | pi_gpu |
thousands of device threads on one GPU |
| hybrid | pi_hybrid |
MPI processes, each running OpenMP threads |
Because the random-number generator is identical everywhere and each worker draws its own independent stream, the answer for a given number of darts and seed is the same no matter how the work is divided. Only the time changes. That is the whole point.
Step 1: Make the tutorial modules available
The tutorial ships as a set of environment modules. Point Lmod at them once per session:
module use /lustre/shared/hpcCourses/pi-tutorial/modules
module avail pi-tutorial
You will see three modules. They are mutually exclusive — loading one unloads the others:
pi-tutorial— the CPU rungs (serial, OpenMP, MPI, hybrid).pi-tutorial-cuda— the NVIDIA GPU rung.pi-tutorial-rocm— the AMD GPU rung.
You do not need to load these by hand for the exercise — each job script loads the right one for you. The module use line above is enough.
Step 2: Get your personal copy of the scripts
Run the launcher once:
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
On this first run it does three things: it copies the job scripts into your personal space at $myScratch/hpcCourse/pi-tutorial/slurm, clears any old results, and submits the whole batch of jobs. Watch them with:
squeue --me
Most of the jobs (serial, GPU, multi-node, hybrid) are complete and will run correctly. Two of them — the OpenMP and the MPI jobs — contain a deliberate blank and will not give a useful result yet. Fixing those blanks is your job, and it is the next step.
Step 3: Fill in the two blanks
When you ask Slurm for resources, Slurm sets aside the cores or tasks you requested — but it does not automatically tell your program about them. You have to pass the number along. Slurm makes that easy by exporting the counts as environment variables. Your task is to connect the two.
Open the OpenMP script in your personal copy (use nano if you are not comfortable with vi):
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multithread.slurm
You will find this line:
export OMP_NUM_THREADS=FIXME
The script asked Slurm for cores with --cpus-per-task, and Slurm exported that count as SLURM_CPUS_PER_TASK. Tell OpenMP to use exactly that many threads:
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
Now the MPI script:
nano $myScratch/hpcCourse/pi-tutorial/slurm/run_multiprocess.slurm
Here the blank is the number of processes to launch:
mpirun -np FIXME pi_mpi --total 1e10 --seed 12345
This script asked Slurm for tasks with --ntasks, and Slurm exported that as SLURM_NTASKS. Launch one process per task:
mpirun -np ${SLURM_NTASKS} pi_mpi --total 1e10 --seed 12345
That is the entire editable surface — two lines. Save both files.
Step 4: Launch the full sweep
Run the launcher again:
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
This time it keeps your edited scripts (it never overwrites your copy), clears the old results, and resubmits everything. The launcher sweeps a range of sizes for you automatically — OpenMP from 1 up to 128 threads, MPI from 1 up to 128 processes — plus the GPU, multi-node, and hybrid runs.
Keep an eye on the queue:
squeue --me
Each finished job writes one result line into $myScratch/hpcCourse/pi-tutorial/results. Wait until the queue is empty before moving on.
Step 5: Analyze your results
The analysis script reads every result file, prints a summary table, and writes three plots. It needs matplotlib, which lives in the SciPy bundle:
module load 2024 SciPy-bundle/2024a
python /lustre/shared/hpcCourses/pi-tutorial/analyze.py $myScratch/hpcCourse/pi-tutorial/results
When it finishes you will have three PNG files in your results directory:
walltime_vs_cores.png— how long each run took, against the number of cores.speedup_vs_cores.png— how much faster than the serial run, against the ideal straight line.best_per_approach.png— the single fastest run of each approach, side by side.
Copy them to your laptop (for example with scp) to view them, or open them through the OnDemand file browser.
Results
A correct run produces a table like the one below (trimmed; your exact times will vary by a few percent). Every run threw the same 10 billion darts.
variant nodes cores device wall_s pi pct_err
serial 1 1 cpu 26.008 3.1416052 0.00040
omp 1 1 cpu 26.499 3.1416052 0.00040
omp 1 16 cpu 1.663 3.1415960 0.00011
omp 1 128 cpu 0.235 3.1416075 0.00047
mpi 1 1 cpu 69.676 3.1416052 0.00040
mpi 1 128 cpu 0.555 3.1416075 0.00047
mpi 4 64 cpu 1.090 3.1416039 0.00036
hybrid 4 128 cpu 0.217 3.1416075 0.00047
gpu 1 1 AMD_Instinct_MI210 0.080 3.1415897 0.00009
gpu 1 1 NVIDIA_A100_80GB_PCIe 0.045 3.1415897 0.00009
Two things are worth noticing straight away:
- The serial run took about 26 seconds; the 128-thread OpenMP run took about 0.24 seconds; the A100 GPU finished in about 0.045 seconds. The work was identical — only the time changed.
- Look at the
picolumn. For any given total work and seed, the estimate is identical across serial, OpenMP, MPI, multi-node, and hybrid — for example every 128-core CPU run lands on 3.1416075. Splitting the darts across more workers did not change the answer, because each worker draws its own independent random stream. (The GPU differs in the last few digits because it uses far more streams; it still lands well within the expected error.)
Discussion
The plots tell a tidy story: add workers, get proportionally faster, almost along the ideal line. That is real — but it is important to understand why it looks so clean, and where the limits are.
This problem is unusually friendly. Throwing darts is "embarrassingly parallel": the workers never need to talk to each other until the very end, when their hit-counts are summed. Most real scientific codes are not like this. A fluid-dynamics solver, a molecular-dynamics simulation, or a genome assembler has data dependencies and communication between workers, and that communication eventually caps how much faster you can go (this ceiling is known as Amdahl's law). Do not expect your own code to scale as cleanly as π does — this tutorial shows you the best case, not the typical one.
More workers buy speed, not accuracy. The error of a Monte Carlo estimate shrinks with the number of darts as roughly 1/√N — so to halve the error you must throw four times as many darts. Adding cores lets you throw a fixed number of darts sooner, or throw more darts in the same time, but it never makes a fixed number of darts more accurate. Speed and accuracy are separate knobs.
Compare equal core counts. In the table, the 4-node MPI run (1.09 s) looks slower than the 1-node MPI run (0.56 s). It is not a penalty for crossing nodes — the 4-node run used 64 cores while the 1-node run used 128. When you compare like with like (64 cores on 1 node ≈ 1.11 s versus 64 cores across 4 nodes ≈ 1.09 s) the times are essentially equal: for this workload the network between nodes is not a bottleneck. Always check the core count before reading a comparison as good or bad news.
The measurements are deliberately rough. Each point is a single run with a single seed, and the fastest runs finish in well under a second — far too short to time precisely on a shared machine. That noise is why a step in the curve can occasionally look better than perfect ("super-linear"); it is timing jitter, not magic. A proper benchmark repeats each run several times and reports the median. This tutorial trades that rigour for clarity and a short queue.
A real-world wrinkle. You may notice the single-process MPI run (about 70 s) is markedly slower than the single-thread OpenMP run (about 26 s) for the same work — and that gap persists across the sweep. That is a property of how this particular MPI program starts up and is being investigated; it is not a general rule that MPI is slower than OpenMP. It is a good reminder that how you measure matters as much as what you measure.
Conclusion
You ran one calculation on a single core and on hundreds, on CPUs and on GPUs, on one node and across four — and the answer never changed, only the time. That is the core lesson of parallel computing on a cluster: you choose a parallel model to match the structure of your problem, and the reward is speed, not correctness.
A few takeaways to carry forward:
- If your work is made of independent pieces (parameter sweeps, many input files, Monte Carlo trials), it will parallelise well — threads, processes, or a GPU all help.
- If your work has steps that depend on each other, expect communication to limit your speedup, and measure before assuming more cores will help.
- More cores make things faster; only more samples (or a better method) make a Monte Carlo result more accurate.
To go further, try editing a script to use --per-worker instead of --total and watch the difference between weak and strong scaling, or move on to packaging your own software with the Apptainer tutorials.
Troubleshooting
myScratch is not set— the course environment is not loaded. Load it and try again.- A GPU job sits in the queue (pending) — the NVIDIA cards may be busy; the job will start when one frees up. The AMD GPU run uses a reservation and is the reliable one for the course.
- You edited a script badly and want a clean copy — delete your script folder and re-run the launcher to get fresh copies:
rm -rf "$myScratch/hpcCourse/pi-tutorial/slurm"
bash /lustre/shared/hpcCourses/pi-tutorial/exercise/launch.sh
See also
- Linux Self Assessment — check your command-line readiness.
- Apptainer: Introduction — packaging software for reproducible runs.
- All tutorials · Workshops · Training Materials