Array jobs

SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use an array job to run your job.

Take the following example:

#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt
#SBATCH --time=10
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4000
#SBATCH --array=0-9%4

echo $SLURM_ARRAY_TASK_ID
              

Let's break this down step by step:

#SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt

This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

#SBATCH --array=0-9%4

This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:

#SBATCH --array=3,7-11

will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

echo $SLURM_ARRAY_TASK_ID

This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n.

Two dimensional arrays?

Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then

SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ]

for each job. What if you need two variables to change instead of one?

Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3

As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:

#SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B