Array jobs - Revision history

Prins0891: to to fix code blocks


2023-06-16T07:46:59Z
<code> to <pre> to fix code blocks

				
				
				
				
				
				← Older revision
				Revision as of 09:46, 16 June 2023
				
Line 3:
Line 3:
Take the following example: Take the following example:



<code> <pre>
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
Line 14:
Line 14:
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
                                 
</code> </pre>
Let's break this down step by step: Let's break this down step by step:



<code> <pre>
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
</code> </pre>
This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what. This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.



<code> <pre>
#SBATCH --array=0-9%4 #SBATCH --array=0-9%4
</code> </pre>
This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example: This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
<code> <pre>
#SBATCH --array=3,7-11 #SBATCH --array=3,7-11
</code> </pre>
will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11. will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.



<code> <pre>
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
</code> </pre>
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has. This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.



Line 41:
Line 41:
== Two dimensional arrays? == == Two dimensional arrays? ==
Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then
<code> <pre>
SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ] SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ]
</code> </pre>
for each job. What if you need two variables to change instead of one? for each job. What if you need two variables to change instead of one?



Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93: Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:
<code> <pre>
A=$((93 / 10)) # A = 9 A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3 B=$((93 % 10)) # B = 3
</code> </pre>



As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then: As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:



<code> <pre>
#SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29 #SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5] A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8] B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B mywork $A $B
</code> </pre>



Haars0011 at 07:43, 16 June 2023
2023-06-16T07:43:59Z


				
				
				
				
				
				← Older revision
				Revision as of 09:43, 16 June 2023
				
Line 14:
Line 14:
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
                                 
<code> </code>
Let's break this down step by step: Let's break this down step by step:



Line 20:
Line 20:
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
<code> </code>
This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what. This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.



<code> <code>
#SBATCH --array=0-9%4 #SBATCH --array=0-9%4
<code> </code>
This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example: This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
<code> <code>
#SBATCH --array=3,7-11 #SBATCH --array=3,7-11
<code> </code>
will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11. will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.



<code> <code>
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
<code> </code>
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has. This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.



Line 41:
Line 41:
== Two dimensional arrays? == == Two dimensional arrays? ==
Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then
  <code>
SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ] SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ]
  </code>
for each job. What if you need two variables to change instead of one? for each job. What if you need two variables to change instead of one?



Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93: Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

<code> <code>
A=$((93 / 10)) # A = 9 A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3 B=$((93 % 10)) # B = 3
<code> </code>



As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then: As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:
Line 60:
Line 59:
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8] B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B mywork $A $B
<code> </code>



Haars0011: source to code
2023-06-16T07:39:59Z
source to code

				
				
				
				
				
				← Older revision
				Revision as of 09:39, 16 June 2023
				
Line 3:
Line 3:
Take the following example: Take the following example:



<source lang='bash'> <code>
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
Line 14:
Line 14:
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
                                 
</source> <code>
Let's break this down step by step: Let's break this down step by step:



<source lang='bash'> <code>
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
</source> <code>
This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what. This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.



<source lang='bash'> <code>
#SBATCH --array=0-9%4 #SBATCH --array=0-9%4
</source> <code>
This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example: This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
<source lang='bash'> <code>
#SBATCH --array=3,7-11 #SBATCH --array=3,7-11
</source> <code>
will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11. will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.



<source lang='bash'> <code>
echo $SLURM_ARRAY_TASK_ID echo $SLURM_ARRAY_TASK_ID
</source> <code>
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has. This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.



Line 48:
Line 48:
Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93: Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:



<source lang='bash'> <code>
A=$((93 / 10)) # A = 9 A=$((93 / 10)) # A = 9
B=$((93 % 10)) # B = 3 B=$((93 % 10)) # B = 3
</source> <code>



As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then: As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:



<source lang='bash'> <code>
#SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29 #SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5] A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8] B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B mywork $A $B
</source> <code>



Dawes001 at 14:58, 15 July 2019
2019-07-15T14:58:59Z


				
				
				
				
				
				← Older revision
				Revision as of 16:58, 15 July 2019
				
Line 6:
Line 6:
#SBATCH --output=output_%A.%a.txt #SBATCH --output=output_%A.%a.txt
#SBATCH --error=error_%A.%a.txt #SBATCH --error=error_%A.%a.txt
#SBATCH --partition=MYPARTITION 
#SBATCH --time=10 #SBATCH --time=10
#SBATCH --ntasks=1 #SBATCH --ntasks=1



Dawes001 at 10:00, 6 September 2017
2017-09-06T10:00:13Z


				
				
				
				
				
				← Older revision
				Revision as of 12:00, 6 September 2017
				
Line 60:
Line 60:
A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5] A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8] B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
mywork $A $B
</source> </source>



Dawes001 at 09:59, 6 September 2017
2017-09-06T09:59:32Z


				
				
				
				
				
				← Older revision
				Revision as of 11:59, 6 September 2017
				
Line 38:
Line 38:
This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has. This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.



So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n. You can So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n.



== Two dimensional arrays? == == Two dimensional arrays? ==



Dawes001: Created page with "SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use a..."
2017-09-06T09:59:08Z
Created page with "SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use a..."
New page
SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use an array job to run your job.



Take the following example:



<source lang='bash'>

#SBATCH --output=output_%A.%a.txt

#SBATCH --error=error_%A.%a.txt

#SBATCH --partition=MYPARTITION

#SBATCH --time=10

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=1

#SBATCH --mem-per-cpu=4000

#SBATCH --array=0-9%4



echo $SLURM_ARRAY_TASK_ID

              

</source>

Let's break this down step by step:



<source lang='bash'>

#SBATCH --output=output_%A.%a.txt

#SBATCH --error=error_%A.%a.txt

</source>

This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.



<source lang='bash'>

#SBATCH --array=0-9%4

</source>

This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:

<source lang='bash'>

#SBATCH --array=3,7-11

</source>

will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.



<source lang='bash'>

echo $SLURM_ARRAY_TASK_ID

</source>

This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.



So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n. You can



== Two dimensional arrays? ==

Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then



SLURM_ARRAY_TASK_ID=[ 0  1  2  3  4  5  6  7  8  9 ]



for each job. What if you need two variables to change instead of one?



Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:



<source lang='bash'>

A=$((93 / 10)) # A = 9

B=$((93 % 10)) # B = 3

</source>



As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:



<source lang='bash'>

#SBATCH --array=0-29   ## 5*6 entries, thus 30, including 0 this is 0-29

A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]

B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]

</source>

← Older revision		Revision as of 09:46, 16 June 2023
Line 3:		Line 3:
	Take the following example:		Take the following example:

	<~~code~~>		<pre>
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
Line 14:		Line 14:
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID

	</~~code~~>		</pre>
	Let's break this down step by step:		Let's break this down step by step:

	<~~code~~>		<pre>
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
	</~~code~~>		</pre>
	This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.		This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

	<~~code~~>		<pre>
	#SBATCH --array=0-9%4		#SBATCH --array=0-9%4
	</~~code~~>		</pre>
	This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:		This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
	<~~code~~>		<pre>
	#SBATCH --array=3,7-11		#SBATCH --array=3,7-11
	</~~code~~>		</pre>
	will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.		will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

	<~~code~~>		<pre>
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID
	</~~code~~>		</pre>
	This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.		This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

Line 41:		Line 41:
	== Two dimensional arrays? ==		== Two dimensional arrays? ==
	Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then		Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then
	<~~code~~>		<pre>
	SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]		SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]
	</~~code~~>		</pre>
	for each job. What if you need two variables to change instead of one?		for each job. What if you need two variables to change instead of one?

	Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:		Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:
	<~~code~~>		<pre>
	A=$((93 / 10)) # A = 9		A=$((93 / 10)) # A = 9
	B=$((93 % 10)) # B = 3		B=$((93 % 10)) # B = 3
	</~~code~~>		</pre>

	As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:		As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:

	<~~code~~>		<pre>
	#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29		#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29
	A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]		A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
	B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]		B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
	mywork $A $B		mywork $A $B
	</~~code~~>		</pre>

← Older revision		Revision as of 09:43, 16 June 2023
Line 14:		Line 14:
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID

	<code>		</code>
	Let's break this down step by step:		Let's break this down step by step:

Line 20:		Line 20:
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
	<code>		</code>
	This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.		This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

	<code>		<code>
	#SBATCH --array=0-9%4		#SBATCH --array=0-9%4
	<code>		</code>
	This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:		This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
	<code>		<code>
	#SBATCH --array=3,7-11		#SBATCH --array=3,7-11
	<code>		</code>
	will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.		will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

	<code>		<code>
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID
	<code>		</code>
	This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.		This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

Line 41:		Line 41:
	== Two dimensional arrays? ==		== Two dimensional arrays? ==
	Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then		Running an array such as above will result in a one dimensional string of jobs, for example, with --array=0-9, then
			<code>
	SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]		SLURM_ARRAY_TASK_ID=[ 0 1 2 3 4 5 6 7 8 9 ]
			</code>
	for each job. What if you need two variables to change instead of one?		for each job. What if you need two variables to change instead of one?

	Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:		Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

	<code>		<code>
	A=$((93 / 10)) # A = 9		A=$((93 / 10)) # A = 9
	B=$((93 % 10)) # B = 3		B=$((93 % 10)) # B = 3
	<code>		</code>

	As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:		As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:
Line 60:		Line 59:
	B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]		B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
	mywork $A $B		mywork $A $B
	<code>		</code>

← Older revision		Revision as of 09:39, 16 June 2023
Line 3:		Line 3:
	Take the following example:		Take the following example:

	<~~source lang='bash'~~>		<code>
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
Line 14:		Line 14:
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID

	<~~/source~~>		<code>
	Let's break this down step by step:		Let's break this down step by step:

	<~~source lang='bash'~~>		<code>
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
	<~~/source~~>		<code>
	This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.		This makes sure your job outputs to a file called output_<Jobnumber>.<Arrayid>.txt, allowing you to track which array ID returned what.

	<~~source lang='bash'~~>		<code>
	#SBATCH --array=0-9%4		#SBATCH --array=0-9%4
	<~~/source~~>		<code>
	This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:		This defined the array job itself. This specifies to run ten jobs, with array id's of 0 to 9, but not to allow more than 4 to run at once. The syntax for this allows you to specify exactly what ID's to use, for example:
	<~~source lang='bash'~~>		<code>
	#SBATCH --array=3,7-11		#SBATCH --array=3,7-11
	<~~/source~~>		<code>
	will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.		will only run array tasks with ID's of 3, 7, 8, 9, 10 and 11.

	<~~source lang='bash'~~>		<code>
	echo $SLURM_ARRAY_TASK_ID		echo $SLURM_ARRAY_TASK_ID
	<~~/source~~>		<code>
	This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.		This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

Line 48:		Line 48:
	Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:		Well, there's a simple function called modulo that can solve this. Let's use an example with a modulo of 10, and an example number of 93:

	<~~source lang='bash'~~>		<code>
	A=$((93 / 10)) # A = 9		A=$((93 / 10)) # A = 9
	B=$((93 % 10)) # B = 3		B=$((93 % 10)) # B = 3
	<~~/source~~>		<code>

	As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:		As you can see, this splits the number in half, allowing a job array of 0-99 to be made into two variables, traversing a 2D array. Bear in mind this always starts at 0, so if you need, say, A to be 1-5, and B to be 3-8, then:

	<~~source lang='bash'~~>		<code>
	#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29		#SBATCH --array=0-29 ## 5*6 entries, thus 30, including 0 this is 0-29
	A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]		A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
	B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]		B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
	mywork $A $B		mywork $A $B
	<~~/source~~>		<code>

← Older revision		Revision as of 16:58, 15 July 2019
Line 6:		Line 6:
	#SBATCH --output=output_%A.%a.txt		#SBATCH --output=output_%A.%a.txt
	#SBATCH --error=error_%A.%a.txt		#SBATCH --error=error_%A.%a.txt
	~~#SBATCH --partition=MYPARTITION~~
	#SBATCH --time=10		#SBATCH --time=10
	#SBATCH --ntasks=1		#SBATCH --ntasks=1

← Older revision		Revision as of 12:00, 6 September 2017
Line 60:		Line 60:
	A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]		A=$((SLURM_ARRAY_TASK_ID/5+1)) # A = [0-4]+1 = [1-5]
	B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]		B=$((SLURM_ARRAY_TASK_ID%6+3)) # B = [0-5]+3 = [3-8]
			mywork $A $B
	</source>		</source>

← Older revision		Revision as of 11:59, 6 September 2017
Line 38:		Line 38:
	This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.		This will print to stdout (and get redirected to output_%A.%a.txt) the environment variable set by SLURM that indicates which Array ID this process has.

	So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n. ~~You can~~		So, once this job is run, we will end up with ten files, all called output_<jobid>.<n>.txt, containing the number n.

	== Two dimensional arrays? ==		== Two dimensional arrays? ==

Array jobs - Revision history

Prins0891: to to fix code blocks

Haars0011 at 07:43, 16 June 2023

Haars0011: source to code

Dawes001 at 14:58, 15 July 2019

Dawes001 at 10:00, 6 September 2017

Dawes001 at 09:59, 6 September 2017

Dawes001: Created page with "SLURM can simplify your efforts if you are planning on submitting multiple independent jobs in parallel. Rather than having to use sbatch multiple times, you can instead use a..."

Prins0891: `to to fix code blocks`