Haars0011: IA migration §8: new Performance Optimization/Multiple nodes (arrayjobs) (array jobs as scaling technique) (via create-page on MediaWiki MCP Server)

2026-06-18T13:03:42Z

IA migration §8: new Performance Optimization/Multiple nodes (arrayjobs) (array jobs as scaling technique) (via create-page on MediaWiki MCP Server)

New page

When you have many independent tasks to run — the same program over many inputs, parameter values, or samples — a [[Array Jobs|job array]] is usually the best way to scale out. Instead of one big parallel program, you submit one array of many small jobs, and the scheduler spreads them across the cluster as resources free up.

This is "embarrassingly parallel" work: the tasks do not need to talk to each other (unlike [[Performance Optimization/Multiple nodes (MPI)|MPI]]). It is often the simplest and most effective way to use many nodes at once.

== When to use a job array ==

* You run the same analysis over many files or samples.
* You sweep a parameter over many values.
* Each task is independent and can run on its own.

== How it works ==

A job array is a single submission with many tasks, each with its own <code>$SLURM_ARRAY_TASK_ID</code> that selects which input it processes. The scheduler runs as many tasks at once as there is room for and queues the rest. Because each task is a separate job, an array naturally spreads across many nodes without any parallel programming — and if some tasks fail, you can rerun just those.

For the syntax and worked examples, see [[Array Jobs]].

== See also ==
* [[Array Jobs]]
* [[Performance Optimization/Multiple CPUs]]
* [[Performance Optimization/Multiple nodes (MPI)]]
* [[Scheduler Overview (Slurm)]]

Performance Optimization/Multiple nodes (arrayjobs) - Revision history

Haars0011: IA migration §8: new Performance Optimization/Multiple nodes (arrayjobs) (array jobs as scaling technique) (via create-page on MediaWiki MCP Server)