https://wiki.anunna.wur.nl/index.php?action=history&feed=atom&title=Debugging_JobsDebugging Jobs - Revision history2026-06-19T08:17:54ZRevision history for this page on the wikiMediaWiki 1.45.3https://wiki.anunna.wur.nl/index.php?title=Debugging_Jobs&diff=2867&oldid=prevHaars0011: IA migration §8: new Debugging Jobs page (via create-page on MediaWiki MCP Server)2026-06-18T14:22:05Z<p>IA migration §8: new Debugging Jobs page (via create-page on MediaWiki MCP Server)</p>
<p><b>New page</b></p><div>When a job fails, exits early, or produces the wrong results, a few systematic checks usually find the cause. This page covers how to work out what went wrong; for the monitoring commands themselves, see [[Monitoring Jobs]].<br />
<br />
== Start with the logs ==<br />
<br />
SLURM writes your job's standard output and error to the files you set with <code>--output</code> and <code>--error</code> in your batch script. Read these first — most failures (a missing file, a typo, an out-of-memory message, a module that was not loaded) show up there.<br />
<br />
If you did not set those options, the output goes to <code>slurm-<jobid>.out</code> in the directory you submitted from.<br />
<br />
== Check how the job ended ==<br />
<br />
For a finished job, <code>sacct</code> shows its exit code and state (COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY, CANCELLED):<br />
<br />
<syntaxhighlight lang="bash"><br />
sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,MaxRSS,ReqMem<br />
</syntaxhighlight><br />
<br />
Common patterns:<br />
<br />
* '''TIMEOUT''' — the job hit its <code>--time</code> limit. Request more time, or make the work faster or smaller.<br />
* '''OUT_OF_MEMORY''' — the job needed more memory than it requested. Increase <code>--mem</code> (or <code>--mem-per-cpu</code>); compare <code>MaxRSS</code> against <code>ReqMem</code> to size it.<br />
* '''FAILED''' with a non-zero exit code — the program itself errored; check the logs.<br />
<br />
See [[Monitoring Jobs]] for more on <code>sacct</code> and the other monitoring tools.<br />
<br />
== Reproduce it interactively ==<br />
<br />
If the logs are not enough, reproduce the problem in an [[Interactive Jobs|interactive job]]. Request the same resources, then run the commands from your script by hand and watch what happens. This is the quickest way to debug module loading, paths, and input files.<br />
<br />
== Check a running job ==<br />
<br />
For a job that is running but behaving oddly (too slow, using too much memory), inspect its live resource use with <code>sstat</code>, or connect to the node it runs on — see [[Compute Nodes]] and [[Monitoring Jobs]].<br />
<br />
== Common causes ==<br />
<br />
* Software not available because a [[Environment Modules|module]] (or its bucket) was not loaded in the script.<br />
* Relative paths that worked interactively but not from the job's working directory.<br />
* Running heavy work on, or writing large amounts of I/O to, the wrong filesystem — see [[Storage Systems Overview]].<br />
* Asking for resources the partition cannot provide — see [[Partitions / Queues]].<br />
<br />
== See also ==<br />
* [[Monitoring Jobs]]<br />
* [[Interactive Jobs]]<br />
* [[Batch Jobs]]<br />
* [[Scheduler Overview (Slurm)]]</div>Haars0011