Haars0011: Phase 1 § 4 P1.4.7: merge Using Slurm § Monitoring + Monitoring job execution + SACCT + Node usage graph into Monitoring Jobs (via create-page on MediaWiki MCP Server)

2026-06-16T09:41:49Z

Phase 1 § 4 P1.4.7: merge Using Slurm § Monitoring + Monitoring job execution + SACCT + Node usage graph into Monitoring Jobs (via create-page on MediaWiki MCP Server)

New page

Once you have submitted a job, you usually want to keep an eye on it: is it queued or running, on which node, using how much memory, producing the output you expected. SLURM exposes this through a small collection of commands; each one answers a different question, and together they cover everything from "is it running" to "what did it cost me last month".

This page is a reference for the four common needs:

* '''Is the job in the queue, and where in the queue?''' — <code>squeue</code> and <code>scontrol</code>.
* '''Is the running job doing what I want?''' — the output streams from your script, plus <code>sstat</code> or <code>top</code> for live resource usage.
* '''What did finished jobs actually use?''' — <code>sacct</code>.
* '''How busy is the cluster overall?''' — <code>node_usage_graph</code>.

For submitting jobs see [[Batch Jobs]] and [[Interactive Jobs]]; to stop a job, see [[Cancelling Jobs]].

== Listing your jobs ==

<code>squeue</code> shows every job currently known to the scheduler. The most common form is just:

<syntaxhighlight lang="bash">
squeue
</syntaxhighlight>

To restrict it to your own jobs:

<syntaxhighlight lang="bash">
squeue -u $USER
</syntaxhighlight>

The <code>-l</code> flag adds the time limit set on each job (useful for spotting a misconfigured time request):

<syntaxhighlight lang="bash">
squeue -l -j <jobid>
</syntaxhighlight>

== Detailed information for one job ==

<code>scontrol show jobid</code> dumps everything SLURM knows about a job — start time, allocated nodes, command line, partition, account, exit code:

<syntaxhighlight lang="bash">
scontrol show jobid <jobid>
</syntaxhighlight>

This works for jobs that are currently in the system (running, pending, completing). Once a job has cleared the queue, use <code>sacct</code> (below) to get its accounting record.

== When will a pending job start? ==

If <code>squeue</code> shows your job as <code>PD</code> (pending) with a reason like <code>ReqNodeNotAvail</code>, SLURM has an estimate for when it can start:

<syntaxhighlight lang="bash">
squeue --start -j <jobid>
</syntaxhighlight>

The <code>START_TIME</code> column is an estimate — it can shift forward or backward as other jobs finish or new jobs arrive — but it is the best signal SLURM has.

== Tracking output streams ==

The output and error files declared in your batch script (see [[Batch Jobs]]) are written in real time. The most useful tool to follow them is <code>tail</code>:

<syntaxhighlight lang="bash">
tail -f output_<jobid>.txt
</syntaxhighlight>

This keeps the file open and prints new lines as your job writes them. Press <code>Ctrl-C</code> to stop following. To see only the last N lines instead of following:

<syntaxhighlight lang="bash">
tail -n <N> output_<jobid>.txt
</syntaxhighlight>

For a long output file you want to page through, use <code>less</code>:

<syntaxhighlight lang="bash">
less output_<jobid>.txt
</syntaxhighlight>

Press <code>q</code> to quit.

== Live resource usage ==

The output streams tell you what your code is logging; they don't tell you what it is actually using. For CPU load and memory while a job is running, you have two options.

=== sstat ===

<code>sstat</code> reports instantaneous resource usage. To use it your batch script needs to launch the work step under <code>srun</code> (so that SLURM tracks it as a job step). Example final line of the script:

<syntaxhighlight lang="bash">
srun python3 calc_pi.py
</syntaxhighlight>

Then, with the job ID from the submission, query a few useful fields:

<syntaxhighlight lang="bash">
sstat --format=AveCPU,AveRSS,MaxRSS -P -j <jobid>
</syntaxhighlight>

The full list of available fields is in [https://slurm.schedmd.com/sstat.html the sstat manual].

=== Logging top inside the job ===

If you want a continuous record of what the OS sees while the job runs, run <code>top</code> alongside your work and let its output land in the job's stdout. Add this near the start of the script (replace the username):

<syntaxhighlight lang="bash">
srun --overcommit --ntasks=1 top -b -u user001 &
python3 calc_pi.py
</syntaxhighlight>

The <code>&</code> backgrounds <code>top</code> so the real workload starts immediately. <code>top -b</code> runs in batch mode (no terminal control) and writes one snapshot every three seconds. The result in your output file looks like:

<syntaxhighlight lang="text">
top - 18:09:12 up 53 days, 22:53, 0 users, load average: 27,04, 27,63, 26,53
Tasks: 1068 total, 4 running, 1064 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41,9 us, 0,2 sy, 0,0 ni, 57,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29260 user001 20 0 15,6g 15,2g 17392 R 1777 1,5 4014:16 R
29447 user001 20 0 178604 12900 1676 R 1,6 0,0 6:33.17 top
</syntaxhighlight>

== Historical accounting with sacct ==

<code>sacct</code> reports on jobs that have left the queue. It reads from the accounting database, so it works for both finished and still-running jobs, including ones from months back.

The simplest form lists today's jobs:

<syntaxhighlight lang="bash">
sacct -a
</syntaxhighlight>

For a real report — for instance an entire month, with the fields you actually care about, written out as CSV — pass <code>--format</code>, a date range, and the parsable flags:

<syntaxhighlight lang="bash">
sacct -P -X --delimiter=',' \
-S 2026-01-01 -E 2026-02-01 \
--format=comment%15,User,Partition%20,JobID,JobName,ncpus,nnodes,NodeList,Start,alloccpus,cputime%12,cputimeraw,state \
> usage_report_$(date -I).csv
</syntaxhighlight>

The flags break down as:

* <code>-P</code> writes parsable output (no column alignment, just delimited fields).
* <code>-X</code> hides the per-step rows and keeps just one row per job.
* <code>-S</code> / <code>-E</code> set the start and end of the window. Without these <code>sacct</code> only looks at today.
* <code>--format</code> selects the columns; the <code>%NN</code> suffix sets a width.
* <code>cputimeraw</code> is the field you usually want for cost analysis — total CPU-seconds.

The resulting CSV opens directly in Excel or any spreadsheet. The full field list is in [https://slurm.schedmd.com/sacct.html the sacct manual].

== Cluster-wide view: node_usage_graph ==

To see how busy the cluster as a whole is — which nodes are crunching, which are drained, which are reserved — use <code>node_usage_graph</code>. It is a small wrapper around <code>sacct</code> that renders a per-node bar chart in your terminal.

<syntaxhighlight lang="bash">
module load legacy
module load anunna
node_usage_graph
</syntaxhighlight>

The output is one row per node. Each row contains up to two strips of characters: the top strip shows CPU activity, the bottom shows memory:

<syntaxhighlight lang="text">
node: |0% 100%|
fat002: CCCCCCCCC
MMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
node003:CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
MM
node010:
node040:DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
node042:RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
</syntaxhighlight>

Letters mean:

* <code>C</code> — CPU reserved and in use
* <code>c</code> — CPU reserved but idle
* <code>M</code> — Memory reserved and in use
* <code>m</code> — Memory reserved but unused
* <code>D</code> — Drained node (unavailable for new jobs)
* <code>R</code> — Reserved node (held for a specific user or event — see [[Reservations]])
* <code>P</code> — Powered off (energy saving)

<code>node_usage_graph</code> shows current allocation but not the queue depth. For "how busy is the queue?" stick with <code>squeue</code>.

== Job status codes ==

<code>squeue</code> shows a two-letter code in the <code>ST</code> column. The most common are <code>R</code> (running), <code>PD</code> (pending), and <code>CG</code> (completing). The full list:

{| class="wikitable"
|-
! Code !! State !! Description
|-
| <code>CA</code> || CANCELLED || Job was explicitly cancelled by the user or a system administrator. The job may or may not have been initiated.
|-
| <code>CD</code> || COMPLETED || Job has terminated all processes on all nodes.
|-
| <code>CF</code> || CONFIGURING || Job has been allocated resources, but is waiting for them to become ready for use (e.g. booting).
|-
| <code>CG</code> || COMPLETING || Job is in the process of completing. Some processes on some nodes may still be active.
|-
| <code>F</code> || FAILED || Job terminated with a non-zero exit code or other failure condition.
|-
| <code>NF</code> || NODE_FAIL || Job terminated due to failure of one or more allocated nodes.
|-
| <code>PD</code> || PENDING || Job is awaiting resource allocation.
|-
| <code>R</code> || RUNNING || Job currently has an allocation.
|-
| <code>S</code> || SUSPENDED || Job has an allocation, but execution has been suspended.
|-
| <code>TO</code> || TIMEOUT || Job terminated upon reaching its time limit.
|}

== See also ==

* [[Batch Jobs]]
* [[Interactive Jobs]]
* [[Cancelling Jobs]]
* [[Reservations]]
* [[Using Slurm | Scheduler overview]]
* [https://slurm.schedmd.com/squeue.html squeue documentation]
* [https://slurm.schedmd.com/scontrol.html scontrol documentation]
* [https://slurm.schedmd.com/sstat.html sstat documentation]
* [https://slurm.schedmd.com/sacct.html sacct documentation]

Monitoring Jobs - Revision history

Haars0011: Phase 1 § 4 P1.4.7: merge Using Slurm § Monitoring + Monitoring job execution + SACCT + Node usage graph into Monitoring Jobs (via create-page on MediaWiki MCP Server)