Scheduler Overview (Slurm): Difference between revisions
No edit summary |
|||
| (78 intermediate revisions by 9 users not shown) | |||
| Line 1: | Line 1: | ||
The resource allocation | The resource allocation and scheduling software on Anunna is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement. This page is the entry point — most topics have their own page; below is a short summary plus links. | ||
== | == What's on which page == | ||
* [[Partitions / Queues]] — list of partitions (<code>main</code>, <code>gpu</code>, <code>gpu_amd</code>) and how to choose one. | |||
* [[Choosing a node (constraints)]] — defaults, hardware constraints, GPU selection. | |||
< | * [[Batch Jobs]] — writing sbatch scripts and submitting them, including multi-job submissions and dependencies. | ||
* [[Interactive Jobs]] — <code>sinteractive</code> and <code>salloc</code> for live shell sessions on a compute node. | |||
* [[Array jobs]] — running the same script many times with a varying parameter. | |||
* [[Monitoring Jobs]] — <code>squeue</code>, <code>scontrol</code>, <code>sstat</code>, <code>sacct</code>, <code>node_usage_graph</code>. | |||
* [[Cancelling Jobs]] — <code>scancel</code>. | |||
* [[Reservations]] — booking nodes in advance for events. | |||
</ | |||
=== | == Quality of Service == | ||
When submitting a job, you may optionally assign a different Quality of Service (QoS) to it: | |||
<syntaxhighlight lang="bash"> | |||
#SBATCH --qos=std | |||
< | </syntaxhighlight> | ||
#SBATCH -- | |||
The QoS values configured on Anunna: | |||
* '''std''' (priority 10) — the default. Use this unless you have a specific reason to pick another. | |||
* '''low''' (priority 1) — reduced priority, but limited to 8 hours per job so a flood of low-priority jobs cannot lock up the cluster. | |||
* '''high''' (priority 20) — higher priority than <code>std</code>. More expensive — see [[Tariffs]]. | |||
* '''interactive''' (priority 100) — the highest priority, exclusively for immediate-running interactive jobs. You may not submit many or large jobs at this QoS. | |||
</ | |||
Jobs can in principle be restarted and rescheduled if a higher-priority job needs cluster resources, but at the time of writing this preemption is not actually configured. | |||
== | == Fair share == | ||
One of the jobs of the cluster job scheduler (SLURM) is to distribute the resources among all the user requesting them. Besides trying to fit jobs in nodes where resources are available for a job, it also needs to keep track of how many jobs a user has started, with how much resources. | |||
Otherwise somebody who has a lot of very small jobs can slowly claim the whole cluster whenever there is a resource available for those small jobs, thus blocking jobs of other people that need more resources. | |||
The way it does this, is by using a fair share for every user. If you have used a lot of resources in the past, your next jobs will be less likely to be scheduled than jobs of people who have just started their jobs. | |||
For more info see https://slurm.schedmd.com/fair_tree.html | |||
== | == Running MPI jobs == | ||
For multi-node MPI workloads see [[MPI on B4F cluster | MPI on Anunna]]. | |||
== | == See also == | ||
* [[Partitions / Queues]] | |||
* [[Choosing a node (constraints)]] | |||
* [[Batch Jobs]] | |||
* [[Interactive Jobs]] | |||
* [[Array jobs]] | |||
* [[Monitoring Jobs]] | |||
* [[Cancelling Jobs]] | |||
* [[Reservations]] | |||
* [[Tariffs | Costs associated with resource usage]] | |||
== | == External links == | ||
* [http://slurm.schedmd.com Slurm official documentation] | * [http://slurm.schedmd.com Slurm official documentation] | ||
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia] | * [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia] | ||
Latest revision as of 14:25, 6 November 2026
The resource allocation and scheduling software on Anunna is SLURM: Simple Linux Utility for Resource Management. This page is the entry point — most topics have their own page; below is a short summary plus links.
What's on which page
- Partitions / Queues — list of partitions (
main,gpu,gpu_amd) and how to choose one. - Choosing a node (constraints) — defaults, hardware constraints, GPU selection.
- Batch Jobs — writing sbatch scripts and submitting them, including multi-job submissions and dependencies.
- Interactive Jobs —
sinteractiveandsallocfor live shell sessions on a compute node. - Array jobs — running the same script many times with a varying parameter.
- Monitoring Jobs —
squeue,scontrol,sstat,sacct,node_usage_graph. - Cancelling Jobs —
scancel. - Reservations — booking nodes in advance for events.
Quality of Service
When submitting a job, you may optionally assign a different Quality of Service (QoS) to it:
#SBATCH --qos=std
The QoS values configured on Anunna:
- std (priority 10) — the default. Use this unless you have a specific reason to pick another.
- low (priority 1) — reduced priority, but limited to 8 hours per job so a flood of low-priority jobs cannot lock up the cluster.
- high (priority 20) — higher priority than
std. More expensive — see Tariffs. - interactive (priority 100) — the highest priority, exclusively for immediate-running interactive jobs. You may not submit many or large jobs at this QoS.
Jobs can in principle be restarted and rescheduled if a higher-priority job needs cluster resources, but at the time of writing this preemption is not actually configured.
Fair share
One of the jobs of the cluster job scheduler (SLURM) is to distribute the resources among all the user requesting them. Besides trying to fit jobs in nodes where resources are available for a job, it also needs to keep track of how many jobs a user has started, with how much resources.
Otherwise somebody who has a lot of very small jobs can slowly claim the whole cluster whenever there is a resource available for those small jobs, thus blocking jobs of other people that need more resources.
The way it does this, is by using a fair share for every user. If you have used a lot of resources in the past, your next jobs will be less likely to be scheduled than jobs of people who have just started their jobs.
For more info see https://slurm.schedmd.com/fair_tree.html
Running MPI jobs
For multi-node MPI workloads see MPI on Anunna.
See also
- Partitions / Queues
- Choosing a node (constraints)
- Batch Jobs
- Interactive Jobs
- Array jobs
- Monitoring Jobs
- Cancelling Jobs
- Reservations
- Costs associated with resource usage