View source for Workflows/Parallel-Computing

Your analysis runs. It just runs for too long, or it no longer fits on one machine. '''Parallel computing''' may be the answer to both, and this page is the map: what it actually means on Anunna, and which of the six workflow types below is yours.

Each section is a short paragraph and a link. Read them, recognise your own problem in one of them, and follow it. If you would rather learn by doing, the [[Tutorials/Parallelism-Estimating-Pi|π estimation tutorial]] walks the same ladder with real code and real timings.

== Why bother ==

Nothing glamorous. The reasons are almost always one of these:

* '''The run takes too long.''' Four days for one analysis means four days before you find out whether the parameters were right.
* '''The dataset outgrew the machine.''' What fitted comfortably on a laptop last year now needs more memory than a laptop has.
* '''There are simply a lot of runs.''' Two hundred samples, one pipeline, one after another after another.
* '''The hardware is sitting idle.''' Anunna's compute nodes carry anywhere from tens to hundreds of cores each, depending on which generation you land on (see [[Compute Hardware Overview]]). A serial job uses exactly one of them and leaves the rest with nothing to do.

A cluster does not make your program faster. It lets '''many pieces of the work happen at once''' — but only if the work can be divided, and only if your program has been told to divide it. Slurm hands you the cores; it does not teach your software to use them.

== The one idea ==

Underneath all the jargon there are only two ways to divide work:

* '''Split the data.''' Every worker gets a different slice of the same job — different samples, different rows, different darts.
* '''Split the steps.''' Every worker takes a different part of the calculation, and they exchange results as they go.

Everything below is a variation on one of those two. What differs is ''who the workers are'' — threads, processes, GPU device threads — and ''how much they have to talk to each other''. That second part is what separates the easy cases from the hard ones, so it is worth noticing which side of it your problem sits on.

== Why not just ask for a thousand cores ==

Because you would not get a thousandfold speedup, and usually not anything close to it.

Every job has parts that cannot be divided: reading the input, setting up, writing the results, and any step that has to wait for the step before it. Adding workers shrinks the part that ''can'' be split and does nothing whatsoever to the part that cannot. Push far enough and the stubborn part is all that is left, and it sets a floor on your wall-clock time no matter how many cores you requested. This is '''[https://en.wikipedia.org/wiki/Amdahl%27s_law Amdahl's law]''', and the practical version of it is: '''measure before you ask for more'''.

Reality adds two more brakes. Workers that need to talk to each other spend real time doing it, and that cost grows as you add them. And a job asking for 256 cores waits in the queue until 256 are free — a 16-core job that starts within ten minutes often finishes long before the greedy one has even begun.

If you would rather see the ceiling than read about it, the [[Tutorials/Parallelism-Estimating-Pi#Discussion|π tutorial's discussion]] has the timings and the scaling curves.

== Which workflow is yours ==

Six types, roughly in order of how much you have to think about. You should be able to recognise yourself in one of them.

'''Serial — one process, one core, start to finish.''' This is the baseline, and it is honestly not a parallel workflow at all. It is here because every other type is best described as a change to it: same job script, one line different. Most software you did not specifically choose for its parallelism is serial, and asking Slurm for more cores will not change that — the extra cores sit idle with your name on them. Start here so the other five have something to differ from. Full page: [[Workflows/Serial-Workflows|Serial Workflows]].

'''Embarrassingly parallel — the same program, many times, on different inputs.''' Two hundred samples through one pipeline. A parameter sweep. A hundred simulations that differ only by their random seed. The runs never need to talk to each other, which makes this both the easiest kind of parallelism and by far the most common in biology. Slurm has a purpose-built tool for it, the job array, and it usually needs no change to your program whatsoever. If your problem fits here, you are in luck. Full page: [[Workflows/Embarrassingly-Parallel|Embarrassingly Parallel]].

'''Multi-threaded — one program, many cores, shared memory.''' The program stays a single process but splits its work across several threads, all reading and writing the same data in memory. This is what a tool means when its manual offers <code>--threads 16</code> or <code>-p 8</code>. It is the gentlest real parallelism to adopt, because someone else already did the hard part. Its hard limit is that every thread must live on the same node: you can grow to that node's core count and not one core further. Full page: [[Workflows/Multi-Threaded-Workflows|Multi-Threaded Workflows]].

'''Multi-process — many processes that talk to each other.''' When one node is not enough, the work is split across separate processes, each with its own private memory, passing data over the network as the calculation proceeds. The standard for this is MPI, and it is what lets a single job span the cluster. The catch is that the program must have been written for MPI from the inside; it is not something you can bolt on from the job script. Full page: [[Workflows/Multi-Process-Workflows|Multi-Process Workflows]].

'''Hybrid — processes between nodes, threads within them.''' A few MPI processes per node, each one running a handful of threads. This buys the reach of MPI with less of its communication overhead, and it is how most large simulation codes actually run at scale. The price is two tuning knobs that interact, and arithmetic that has to come out right — processes times threads must match what you asked Slurm for. This is the one people most often get subtly wrong. Full page: [[Workflows/Hybrid-Workflows|Hybrid Workflows]].

'''GPU — hand the heavy arithmetic to an accelerator.''' A GPU runs thousands of very simple workers at once and is spectacular on the narrow class of problems shaped to suit it. Anunna has both NVIDIA and AMD cards, in separate partitions — see [[Compute Hardware Overview]] for what is actually installed. The limitation here is absolute rather than a matter of degree: the software must already have a GPU version. Most software does not, and no Slurm flag will conjure one. Full page: [[Workflows/GPU-Workflows|GPU Workflows]].

== Where to go next ==

Once you know which type is yours, its page shows what the job script looks like and which modules go with it.

If you would rather see all of them side by side on one problem first, work through [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]]. It takes a single calculation — throwing random darts at a dartboard — and runs it serially, with threads, with processes, across several nodes, and on a GPU. The answer never changes. Only the time does, which is exactly the point.

== See also ==

* [[Tutorials/Parallelism-Estimating-Pi|Parallelism: Estimating π]] — the hands-on companion to this page.
* [[Compute Hardware Overview]] — what the nodes and GPUs actually are.
* [[Workflows|Workflows]] — job submission, debugging, and checkpointing.
* [[Linux Basic/Linux Self Assessment|Linux Self Assessment]] — check your command-line readiness first.