Monitoring job execution - Revision history

Duque004: /* Output stream redirection */

2018-08-27T13:26:26Z

Output stream redirection

← Older revision		Revision as of 15:26, 27 August 2018
Line 2:		Line 2:
	== Output stream redirection ==		== Output stream redirection ==

	The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].		The primary way to monitor job execution is through the ''stdout'' and ''stderr'' streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].

	For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:		For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:
Line 14:		Line 14:
	Replacing X by the desired number of lines.		Replacing X by the desired number of lines.

	If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:		If the output file gets too long and you wish to read from the begining you may combine the commands <code>cat</code> and <code>less</code>:

	<code>cat output_987654.txt \| less</code>		<code>cat output_987654.txt \| less</code>

	Use the Q key to exit less.		Use the Q key to exit <code>less</code>.

	== Monitoring resource usage ==		== Monitoring resource usage ==

Duque004: /* Using top */

2018-08-27T13:25:24Z

Using top

← Older revision		Revision as of 15:25, 27 August 2018
Line 38:		Line 38:
	=== Using top ===		=== Using top ===

	Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name):		Another way is to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name):

	<code>srun --overcommit --ntasks=1 top -b -u user001 &		<code>srun --overcommit --ntasks=1 top -b -u user001 &
	python3 calc_pi.py</code>		python3 calc_pi.py</code>

	This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:		This will log the output of <code>top</code> to the sdtout stream file specific in the SLURM script every 3 seconds. Using the <code>tail</code> command you will be able to see logs like:

	<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53		<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53

Duque004: /* Using sstat */

2018-08-27T13:24:15Z

Using sstat

← Older revision		Revision as of 15:24, 27 August 2018
Line 26:		Line 26:
	=== Using sstat ===		=== Using sstat ===

	<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it ~~starting~~ by changing your SLURM script so that your programme ~~starts~~ with the <code>srun</code> command (this should be the last line in the script):		<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it start by changing your SLURM script so that your programme is launched with the <code>srun</code> command (this should be the last line in the script):

	<code>srun python3 calc_pi.py</code>		<code>srun python3 calc_pi.py</code>

Duque004: /* Output stream redirection */

2018-08-27T13:22:52Z

Output stream redirection

← Older revision		Revision as of 15:22, 27 August 2018
Line 4:		Line 4:
	The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].		The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].

	For this purpose the <code>tail</code> command is particularly useful. To ~~continuous~~ follow the output to a text file use:		For this purpose the <code>tail</code> command is particularly useful. To continuously follow the output to a text file use:

	<code>tail -f output_987654.txt</code>		<code>tail -f output_987654.txt</code>

Duque004: Duque004 moved page Monitoring executions to Monitoring job execution

2018-08-23T09:38:24Z

Duque004 moved page Monitoring executions to Monitoring job execution

← Older revision	Revision as of 11:38, 23 August 2018
(No difference)

Duque004 at 09:37, 23 August 2018

2018-08-23T09:37:17Z

← Older revision		Revision as of 11:37, 23 August 2018
Line 4:		Line 4:
	The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].		The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].

	For this purpose the <code>tail command is particularly useful. To continuous follow the output to a text file use:		For this purpose the <code>tail</code> command is particularly useful. To continuous follow the output to a text file use:

	<code>tail -f output_987654.txt</code>		<code>tail -f output_987654.txt</code>
Line 45:		Line 45:
	This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:		This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:

	<~~code~~>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53		<pre>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53
	Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie		Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie
	%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st		%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
Line 56:		Line 56:
	28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+		28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+
	29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun		29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun
	29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</~~code~~>		29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</pre>

Duque004 at 09:36, 23 August 2018

2018-08-23T09:36:34Z

← Older revision		Revision as of 11:36, 23 August 2018
Line 4:		Line 4:
	The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].		The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].

	For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:		For this purpose the <code>tail command is particularly useful. To continuous follow the output to a text file use:

	tail -f output_987654.txt		<code>tail -f output_987654.txt</code>

	To obtain the last X lines of a text file use:		To obtain the last X lines of a text file use:

	tail -n X output_987654.txt		<code>tail -n X output_987654.txt</code>

	Replacing X by the desired number of lines.		Replacing X by the desired number of lines.
Line 16:		Line 16:
	If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:		If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:

	cat output_987654.txt \| less		<code>cat output_987654.txt \| less</code>

	Use the Q key to exit less.		Use the Q key to exit less.
Line 26:		Line 26:
	=== Using sstat ===		=== Using sstat ===

	sstat is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the srun command (this should be the last line in the script):		<code>sstat</code> is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the <code>srun</code> command (this should be the last line in the script):

	srun python3 calc_pi.py		<code>srun python3 calc_pi.py</code>

	Note down the job number. During execution you can then use sstat, passing the job number with the -j flag:		Note down the job number. During execution you can then use <code>sstat</code>, passing the job number with the <code>-j</code> flag:

	sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654		<code>sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654</code>

	sstat can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].		<code>sstat</code> can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].

	=== Using top ===		=== Using top ===

	Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html top] command. This requires adding an extra command to your SLURM script, again using srun (where "user001" should be replaced by your user name):		Another way it to log the output of the [http://man7.org/linux/man-pages/man1/top.1.html <code>top</code>] command. This requires adding an extra command to your SLURM script, again using <code>srun</code> (where "user001" should be replaced by your user name):

	srun --overcommit --ntasks=1 top -b -u user001 &		<code>srun --overcommit --ntasks=1 top -b -u user001 &
	python3 calc_pi.py		python3 calc_pi.py</code>

	This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:		This will log the output of top to the sdtout stream file specific in the SLURM script every 3 seconds. Using the tail command you will be able to see logs like:

	top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53		<code>top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53
	Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie		Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie
	%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st		%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
Line 56:		Line 56:
	28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+		28627 user001 20 0 113184 1520 1260 S 0,0 0,0 0:00.01 slurm_scri+
	29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun		29253 user001 20 0 245096 4792 1976 S 0,0 0,0 0:00.66 srun
	29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun		29357 user001 20 0 36124 688 16 S 0,0 0,0 0:00.00 srun</code>

Duque004 at 09:32, 23 August 2018

2018-08-23T09:32:26Z

@@ Line 2: / Line 2: @@
 == Output stream redirection ==
-The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [[https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file | text files specified in the SLURM script]].
+The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file text files specified in the SLURM script].
 For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:
@@ Line 34: / Line 34: @@
 sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654
-sstat can provide information on many different variables, for more details check [[https://slurm.schedmd.com/sstat.html | the manual]].
+sstat can provide information on many different variables, for more details check [https://slurm.schedmd.com/sstat.html the manual].
-=== Logging the output of top ===
+=== Using top ===

Duque004 at 09:13, 23 August 2018

2018-08-23T09:13:02Z

← Older revision		Revision as of 11:13, 23 August 2018
Line 26:		Line 26:
	=== Using sstat ===		=== Using sstat ===

	sstat is a SLURM tool that can be used to obtain instantaneous		sstat is a SLURM tool that can be used to obtain instantaneous information on resource usage, CPU load, memory, etc. To use it starting by changing your SLURM script so that your programme starts with the srun command (this should be the last line in the script):

	srun python3 calc_pi.py		srun python3 calc_pi.py

	sstat --format=AveCPU,AveRSS,MaxRSS -P -j ~~7466208~~		Note down the job number. During execution you can then use sstat, passing the job number with the -j flag:

			sstat --format=AveCPU,AveRSS,MaxRSS -P -j 987654

			sstat can provide information on many different variables, for more details check [[https://slurm.schedmd.com/sstat.html \| the manual]].

	=== Logging the output of top ===		=== Logging the output of top ===

Duque004: Created page with " == Output stream redirection == The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to https://wiki.hpcagrogenomics.wur..."

2018-08-23T09:07:19Z

Created page with " == Output stream redirection == The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to https://wiki.hpcagrogenomics.wur..."

New page

== Output stream redirection ==

The primary way to monitor job execution is through the stdout and stderr streams. These are redirected to [[https://wiki.hpcagrogenomics.wur.nl/index.php/Creating_sbatch_script#output_.28stderr.2Cstdout.29_directed_to_file | text files specified in the SLURM script]].

For this purpose the `tail`command is particularly useful. To continuous follow the output to a text file use:

tail -f output_987654.txt

To obtain the last X lines of a text file use:

tail -n X output_987654.txt

Replacing X by the desired number of lines.

If the output file gets too long and you wish to read from the begining you may combine the commands cat and less:

cat output_987654.txt | less

Use the Q key to exit less.

== Monitoring resource usage ==

While the output streams may suffice in most cases, certain programmes might not provide much feedback. This could be the case with a programme that rellies on modules that are not verbose. In such situations it is best to monitor resource usage to gauge job execution. Two possible options are described below.

=== Using sstat ===

sstat is a SLURM tool that can be used to obtain instantaneous

srun python3 calc_pi.py

sstat --format=AveCPU,AveRSS,MaxRSS -P -j 7466208

=== Logging the output of top ===