https://wiki.anunna.wur.nl/api.php?action=feedcontributions&feedformat=atom&user=Bohme001HPCwiki - User contributions [en]2026-04-18T01:52:57ZUser contributionsMediaWiki 1.43.1https://wiki.anunna.wur.nl/index.php?title=User:Bohme001&diff=1348User:Bohme0012014-08-28T12:11:01Z<p>Bohme001: </p>
<hr />
<div>Former Linux system administrator at FB-IT Infrastructure<br><br />
Our team is maintaining +200 Linux servers running mainly on RedHat Enterprise Server and some specials on Ubuntu Server.<br><br />
All HPC AgroGenomics hosts are running Scientific Linux version 6</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1347Using Slurm2014-08-12T15:03:29Z<p>Bohme001: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br><br />
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br><br />
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Low queue jobs.<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_High up infinite 12 down* node[043-048,055-060]<br />
ABGC_High up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_High up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Std up infinite 12 down* node[043-048,055-060]<br />
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Std up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Low up infinite 12 down* node[043-048,055-060]<br />
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Low up infinite 44 idle node[001,006-042,049-054]<br />
</source><br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 100MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC_Std<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 100 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=ABGC_Std<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state when there are not enough resources available to this job.<br />
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1345Main Page2014-06-26T13:21:27Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001..node002 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 custom<br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat<br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-IT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-IT reagdring non WUR accounts). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/Our-facilities/Show/High-Performance-Computing-Cluster-HPC.htm CATAgroFood offers a HPC facilty]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1344Main Page2014-06-26T13:14:37Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001..node002 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 custom<br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat<br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1343Main Page2014-06-26T13:14:04Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001..node002 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 custom<br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat<br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1342Main Page2014-06-26T13:12:59Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001..node002 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1<br><br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br><br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br><br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br><br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat<br><br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1341Main Page2014-06-26T13:12:10Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1340Main Page2014-06-26T13:10:46Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001..node002 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1<br> <br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br><br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br><br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br><br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat<br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1339Main Page2014-06-26T13:10:27Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on June 26, 2014:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1338Main Page2014-06-26T13:05:52Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on November 23, 2013:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001, node004, node006..node008, node010, node012..node015, node017, node020, node022, node027..node035, node037, node039, node050, node051, node053 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
node002, node003, node005, node009, node016, node018, node019, node021, node023..node026, node036, node038, node040..node042, node049, node052, node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 3 1 default<br><br />
node011 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br><br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1<br> <br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br><br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br><br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br><br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1337Main Page2014-06-26T13:01:51Z<p>Bohme001: </p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast network connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on November 23, 2013:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
node001, node004, node006..node008, node010, node012..node015, node017, node020, node022, node027..node035, node037, node039, node050, node051, node053 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
node002, node003, node005, node009, node016, node018, node019, node021, node023..node026, node036, node038, node040..node042, node049, node052, node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 3 1 default<br />
node011 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 3 1 default<br />
master1 master2 UP 67.5 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 <br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 5 1 mds<br />
storage01..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2399 MHz 7 1 login<br />
fat001 fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2300 MHz 5 1 fat</code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration, which also will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution.<br />
* The NFS server is a PowerEdge R720XD. The NFS node will also act as a login node, where users log in and compile applications and submit jobs and share each home directory via nfs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using (200GB soft and 210GB hard limit) :<br />
<br />
<source lang='bash'><br />
quota -s<br />
</source><br />
<br />
The NFS is supported through the NFS server (nfs01) that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at one of two main server centres of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-IT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Mailinglist&diff=1336Mailinglist2014-06-05T11:40:20Z<p>Bohme001: </p>
<hr />
<div>== Available mailling lists ==<br />
In order to improve collaboration between HPC users and to provide an easy to use platform in order to exchange ideas, the HPC has a public mailing list service available.<br><br />
More info about the software itself, mailman, can be found on this [http://www.list.org/ website] . <br><br />
<br />
Information about which lists are hosted on the HPC, is available [https://master1.hpcagrogenomics.wur.nl/mailman/listinfo here] <br><br />
We do recommend our HPC users to subscribe to the [https://master1.hpcagrogenomics.wur.nl/mailman/listinfo/hpcag hpcag] list.<br><br />
Via the website you will be able to start the subscription process.</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Mailinglist&diff=1335Mailinglist2014-06-05T11:38:50Z<p>Bohme001: Created page with "== Available mailling lists == In order to improve collaboration between HPC users and to provide an easy to use platform in order to exchange ideas, the HPC has public access..."</p>
<hr />
<div>== Available mailling lists ==<br />
In order to improve collaboration between HPC users and to provide an easy to use platform in order to exchange ideas, the HPC has public accessible mailing list service available. More info about the service software itself, mailman, can be found on this [http://www.list.org/ website] . <br><br />
<br />
Information about which lists are hosted on the HPC can be found [https://master1.hpcagrogenomics.wur.nl/mailman/listinfo here] <br><br />
We do recommend our HPC users to subscribe to the [https://master1.hpcagrogenomics.wur.nl/mailman/listinfo/hpcag hpcag] list. Via the website you will be able to start the subscription process.</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Main_Page&diff=1334Main Page2014-06-05T11:24:16Z<p>Bohme001: /* See also */</p>
<hr />
<div>The [http://www.breed4food.com/en/breed4food.htm Breed4Food] (B4F) cluster is a joint [http://en.wikipedia.org/wiki/High-performance_computing High Performance Compute] (HPC) infrastructure of the [[About_ABGC | Animal Breeding and Genomics Centre]] (WU-Animal Breeding and Genomics and Wageningen Livestock Research) and four major breeding companies: [http://www.cobb-vantress.com Cobb-Vantress], [https://www.crv4all.nl CRV], [http://www.hendrix-genetics.com Hendrix Genetics], and [http://www.topigs.com TOPIGS]. <br />
<br />
== Rationale and Requirements for a new cluster ==<br />
[[File:Breed4food-logo.jpg|thumb|right|200px|The Breed4Food logo]]<br />
The B4F Cluster is, in a way, the 7th pillar of the [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]. While the other six pillars revolve around specific research themes, the Cluster represents a joint infrastructure. The rationale behind the cluster is to enable the increasing computational needs in the field of genetics and genomics research, by creating a joint facility that will generate benefits of scale, thereby reducing cost. In addition, the joint infrastructure is intended to facilitate cross-organisational knowledge transfer. In that capacity, the B4F Cluster acts as a joint (virtual) laboratory where researchers - academic and applied - can benefit from each other's know how. Lastly, the joint cluster, housed at Wageningen University campus, allows retaining vital and often confidential data sources in a controlled environment, something that cloud services such as Amazon Cloud or others usually can not guarantee.<br />
<br />
<br />
{{-}}<br />
<br />
== Process of acquisition and financing ==<br />
<br />
[[File:Signing_CatAgro.png|thumb|left|300px|Petra Caessens, manager operations of CAT-AgroFood, signs the contract of the supplier on August 1st, 2013. Next to her Johan van Arendonk on behalf of Breed4Food.]]<br />
The B4F cluster was financed by [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood]. The [[B4F_cluster#IT_Workgroup | IT-Workgroup]] formulated a set of requirements that in the end were best met by an offer from [http://www.dell.com/learn/nl/nl/rc1078544/hpcc Dell]. [http://www.clustervision.com ClusterVision] was responsible for installing the cluster at the Theia server centre of FB-ICT.<br />
<br />
{{-}}<br />
<br />
== Architecture of the cluster ==<br />
<br />
[[File:Cluster_scheme.png|thumb|right|600px|Schematic overview of the cluster.]]<br />
The new B4F HPC has a classic cluster architecture: state of the art Parallel File System (PSF), headnodes, compute nodes (of varying 'size'), all connected by superfast internet connections (Infiniband). Implementation of the cluster will be done in stages. The initial stage includes a 600TB PFS, 48 slim nodes of 16 cores and 64GB RAM each, and 2 fat nodes of 64 cores and 1TB RAM each. The overall architecture, that include two head nodes in fall-over configuration and an infiniband network backbone, can be easily expanded by adding nodes and expanding the PFS. The cluster management software is designed to facilitate a heterogenous and evolving cluster.<br />
{{-}}<br />
=== Nodes ===<br />
The cluster consists of a bunch of separate machines that each has its own operating system. The default operating system throughout the cluster is [https://www.scientificlinux.org Scientific Linux] version 6. Scientific Linux (SL) is based on [http://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux Red Hat Enterprise Linux (RHEL)], which currently is at version 6. SL therefore follows the versioning scheme of RHEL. <br />
<br />
The cluster has two master nodes in a redundant configuration, which means that if one crashes, the other will take over seamlessly. Various other nodes exist to support the two main file systems (the Lustre parallel file system and the NFS file system). The actual computations are done on the worker nodes or compute nodes. The cluster is configured in a heterogeneous fashion: it consists of 48 so called 'slim nodes', that each have 16 cores and 64GB of RAM (called 'node001' through 'node060'; note that not all node names map to physical nodes), and two so called 'fat nodes' that each have 64 cores and 1TB of RAM ('fat001' and 'fat002').<br />
<br />
Information from the Cluster Management Portal, as it appeared on November 23, 2013:<br />
<code>DEVICE INFORMATION<br />
Hostname State Memory Cores CPU Speed GPU NICs IB Category<br />
master1, master2 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 <br />
node001..node042, node049..node054 UP 67.6 GiB 16 Intel(R) Xeon(R) CPU E5-2660 0+ 1200 MHz 3 1 default<br />
node043..node048, node055..node060 DOWN N/A N/A N/A N/A N/A N/A N/A default<br />
mds01, mds02 UP 16.8 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 5 1 mds<br />
storage01 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2200 MHz 5 1 oss<br />
storage02..storage06 UP 67.6 GiB 32 Intel(R) Xeon(R) CPU E5-2660 0+ 2199 MHz 5 1 oss<br />
nfs01 UP 67.6 GiB 8 Intel(R) Xeon(R) CPU E5-2609 0+ 2400 MHz 7 1 login<br />
fat001, fat002 UP 1.0 TiB 64 AMD Opteron(tm) Processor 6376 2299 MHz 5 1 fat </code><br />
<br />
Main cluster node configuration:<br />
* Master nodes: 2 PowerEdge R720 master nodes in a failover configuration<br />
* The NFS server is a PowerEdge R720XD, which will share some applications and databases with machines in the cluster, for which the parallel file system is not the ideal solution. The NFS node will also act as a login node, where users log in and compile applications and submit jobs.<br />
* 50 compute nodes<br />
** 12x Dell PowerEdge C6000 enclosures, each containing four nodes<br />
** 48x Dell PowerEdge C6220; 16 Intel Xeon cores, 64GB RAM each<br />
** 2x Dell R815; 64 AMD Opteron cores, 1TB RAM each<br />
Hyperthreading is disabled in compute nodes.<br />
<br />
=== Filesystems ===<br />
<br />
[[File:Storage_pic.png|thumb|right|300px|Schematic overview of storage components of the B4F cluster.]]<br />
The B4F Cluster has two primary file systems, each with different properties and purposes.<br />
==== Parallel File System: Lustre ====<br />
At the base of the cluster is an ultrafast file system, a so called [http://en.wikipedia.org/wiki/Parallel_file_system Parallel File System] (PFS). The current size of the PFS is around 600TB. The PFS implemented in the B4F Cluster is called [http://en.wikipedia.org/wiki/Lustre_(file_system) Lustre]. Lustre has become very popular in recent years due to the fact that it is very feature rich, deemed very stable, and is Open Source. Lustre nowadays is the default option for PFS in Dell clusters as well as clusters sold by other vendors. The PFS is mounted on all head nodes and worker nodes of the cluster, providing a seamless integration between compute and data infrastructure. The strength of a PFS is speed - the total I/O should be up to 15GB/s. by design. With a very large number of compute nodes - and with very high volumes of data - these high read-write speeds that the PSF can provide are necessary. The Lustre filesystem is divided in [[Lustre_PFS_layout | several partitions]], each differing in persistence and backup features. The Lustre PSF is meant to store (shared) data that is likely to be used for analysis in the near future. Personal analysis scripts, software, or additional small data files can be stored in the $HOME directory of each of the users.<br />
<br />
The hardware components of the PFS:<br />
* 2x Dell PowerEdge R720<br />
* 1x Dell PowerVault MD3220<br />
* 6x Dell PowerEdge R620<br />
* 6x Dell PowerVault MD3260<br />
<br />
==== Network File System (NFS): $HOME dirs ====<br />
Each user will have his/her own home directory. The path of the home directory will be: <br />
<br />
/home/[name partner]/[username]<br />
<br />
/home lives on a so called [http://en.wikipedia.org/wiki/Network_File_System Network File System], or NFS. The NFS is separate from the PFS and is far more limited in I/O (read/write speeds, latency, etc) than the PFS. This means that it is not meant to store large datavolumes that require high data transfer or small latency. Compared to the Lustre PFS (600TB in size), the size of the NFS is small in comparison - only 20TB. The /home partition will be backed up daily. The amount of space that can be allocated is limited per user. Personal quota and total use per user can be found using:<br />
<br />
<source lang='bash'><br />
quota<br />
</source><br />
<br />
The NFS is supported through the NFS server that also serves as access point to the cluster.<br />
<br />
Hardware components of the NFS:<br />
* 1x Dell PowerEdge R720XD<br />
* 1x Dell PowerVault MD3220<br />
<br />
=== Network ===<br />
The various components - head-nodes, worker nodes, and most importantly, the Lustre PFS - are all interconnected by an ultra-high speed network connection called [http://en.wikipedia.org/wiki/Infiniband InfiniBand]. A total of 7 InifiniBand switches are configured in a [http://en.wikipedia.org/wiki/Fat_tree fat tree] configuration.<br />
<br />
== Housing at Theia ==<br />
[[File:Map_Theia.png|thumb|left|200px|Location of Theia, just outside of Wageningen campus]]<br />
The B4F Cluster is housed at the main server centre of WUR-FB-ICT, near Wageningen Campus. The building (Theia) may not look like much from the outside (used to function as potato storage) but inside is a modern server centre that includes, a.o., emergency power backup systems and automated fire extinguishers. Many of the server facilities provided by FB-ICT that are used on a daily basis by WUR personnel and students are located there, as is the B4F Cluster. Access to Theia is evidently highly restricted and can only be granted in the presence of a representative of FB-ICT.<br />
{{-}}<br />
{| width="90%"<br />
|- valign="top"<br />
| width="10%" |<br />
<br />
| width="30%" |<br />
[[File:Cluster2_pic.png|thumb|left|220px|Some components of the cluster after unpacking.]]<br />
| width="70%" |<br />
[[File:Cluster_pic.png|thumb|right|400px|The final configuration after installation.]]<br />
|}<br />
{{-}}<br />
<br />
== Management ==<br />
<br />
=== Project Leader ===<br />
* Stephen Janssen (Wageningen UR,FB-IT, Service Management)<br />
<br />
=== Daily Project Management ===<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR,FB-IT, Infrastructure)]]<br />
* Andre ten Böhmer (Wageningen UR, FB-ICT, Infrastructure)<br />
<br />
[[Maintenance_and_Management | Maintenance and Management]]<br />
<br />
=== Steering Group ===<br />
Ensures that the HPC generates enough revenues and meets the needs of the users. This includes setting fees, developing contracts, attracting new users, decisions on investments in the HPC and communication. <br />
* Frido Hamoen (CRV, on behalf of Breed4Food industrial partners, replaced Alfred de Vries in August)<br />
* Petra Caessens (CAT-AgroFood)<br />
* Wojtek Sablik (Wageningen UR, FB-IT, Infrastructure)<br />
* Edda Neuteboom (CAT_AgroFood, secretariat)<br />
* Johan van Arendonk (Wageningen UR, chair).<br />
<br />
=== IT Workgroup ===<br />
[[File:Image_(1).jpeg|thumb|right|380px|(part of) the IT working group in front of the B4F Cluster]]<br />
Is responsible for the technical performance of the HPC. The IT-workgroup has been involved in the design of the HPC and the selection of the supplier. They will support the technical management of the HPC and share experiences to ensure that the HPC meets the needs of its users. The IT-workgroup will advise the steering group on investments in software and hardware.<br />
* [[User:Janss115 | Stephen Janssen (Wageningen UR, FB-IT, Service Management)]]<br />
* [[User:pollm001 | Koen Pollmann (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Bohme001 | Andre ten Böhmer (Wageningen UR, FB-IT, Infrastructure)]]<br />
* [[User:Barris01 | Wes Barris (Cobb)]]<br />
* [[User:Vereij01 | Addie Vereijken (Hendrix Genetics)]]<br />
* [[User:dongen01 | Henk van Dongen (Topigs)]]<br />
* Harry Dijkstra (CRV)<br />
* [[User:Calus001 | Mario Calus (ABGC-WLR)]]<br />
* [[User:Megen002 | Hendrik-Jan Megens (ABGC-ABG)]]<br />
{{-}}<br />
<br />
=== User Group ===<br />
The User Group ultimately is the most important of all groups, because it encompasses the users for which the infrastructure was built. In addition, successful use of the cluster will rely on an active community of users that is willing to share knowledge and best practices, including maintenance and expansion of this Wiki. Regular User Group meetings will be held in the future [frequency to be determined] to facilitate this process.<br />
<br />
* [[List_of_users | List of users (alphabetical order)]]<br />
<br />
== Access Policy ==<br />
Access policy is still a work in progress. In principle, all staff and students of the five main partners will have access to the cluster. Access needs to be granted actively (by creation of an account on the cluster by FB-ICT). Use of resources is limited by the scheduler. Depending on availability of queues ('partitions') granted to a user, priority to the system's resources is regulated. <br />
<br />
=== Contact Persons ===<br />
A request to access the cluster needs to be directed to one of the following persons (please refer to appropriate partner):<br />
<br />
==== Cobb-Vantress ====<br />
* Wes Barris<br />
* Jun Chen<br />
<br />
==== ABGC ====<br />
===== Animal Breeding and Genetics =====<br />
* [[User:Hulze001 |Alex Hulzebosch]]<br />
* [[User:Megen002 | Hendrik-Jan Megens]]<br />
<br />
===== Wageningen Livestock Research =====<br />
* Mario Calus<br />
* Ina Hulsegge<br />
==== CRV ====<br />
* Frido Hamoen<br />
* Chris Schrooten<br />
==== Hendrix Genetics ==== <br />
* Ton Dings<br />
* Abe Huisman<br />
* Addie Vereijken<br />
==== Topigs ====<br />
* [[User:dongen01 | Henk van Dongen]]<br />
* Egiel Hanenbarg<br />
* Naomi Duijvensteijn<br />
<br />
== Using the B4F Cluster ==<br />
=== Gaining access to the B4F Cluster ===<br />
Access to the cluster and file transfer are done by [http://en.wikipedia.org/wiki/Secure_Shell ssh-based protocols].<br />
* [[log_in_to_B4F_cluster | Logging into cluster using ssh and file transfer]]<br />
<br />
=== Cluster Management Software and Scheduler ===<br />
The B4F cluster uses Bright Cluster Manager software for overall cluster management, and Slurm as job scheduler.<br />
* [[BCM_on_B4F_cluster | Monitor cluster status with BCM]]<br />
* [[SLURM_on_B4F_cluster | Submit jobs with Slurm]]<br />
* [[SLURM_Compare | Rosetta Stone of Workload Managers]]<br />
<br />
=== Installation of software by users ===<br />
<br />
* [[Domain_specific_software_on_B4Fcluster_installation_by_users | Installing domain specific software: installation by users]]<br />
* [[Setting local variables]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Installed software ===<br />
<br />
* [[Globally_installed_software | Globally installed software]]<br />
* [[ABGC_modules | ABGC specific modules]]<br />
<br />
=== Being in control of Environment parameters ===<br />
<br />
* [[Using_environment_modules | Using environment modules]]<br />
* [[Setting local variables]]<br />
* [[Setting_TMPDIR | Set a custom temporary directory location]]<br />
* [[Installing_R_packages_locally | Installing R packages locally]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
=== Controlling costs ===<br />
<br />
* [[SACCT | using SACCT to see your costs]]<br />
<br />
== Miscellaneous ==<br />
* [[Bioinformatics_tips_tricks_workflows | Bioinformatics tips, tricks, and workflows]]<br />
<br />
== See also ==<br />
* [[Maintenance_and_Management | Maintenance and Management]]<br />
* [[Mailinglist | Electronic mail discussion lists]]<br />
* [[About_ABGC | About ABGC]]<br />
* [[Computer_cluster | High Performance Computing @ABGC]]<br />
* [[Lustre_PFS_layout | Lustre Parallel File System layout]]<br />
<br />
== External links ==<br />
{| width="90%"<br />
|- valign="top"<br />
| width="30%" |<br />
* [http://www.breed4food.com/en/show/Breed4Food-initiative-reinforces-the-Netherlands-position-as-an-innovative-country-in-animal-breeding-and-genomics.htm Breed4Food programme]<br />
* [http://www.wageningenur.nl/en/Expertise-Services/Facilities/CATAgroFood-3/CATAgroFood-3/News-and-agenda/Show/CATAgroFood-invests-in-a-High-Performance-Computing-cluster.htm CATAgroFood invests in HPC]<br />
* [http://www.cobb-vantress.com Cobb-Vantress homepage]<br />
<br />
<br />
| width="30%" |<br />
* [https://www.crv4all.nl CRV homepage]<br />
* [http://www.hendrix-genetics.com Hendrix Genetics homepage]<br />
* [http://www.topigs.com TOPIGS homepage]<br />
| width="30%" |<br />
* [http://en.wikipedia.org/wiki/Scientific_Linux Scientific Linux]<br />
* [http://en.wikipedia.org/wiki/Help:Cheatsheet Help with editing Wiki pages]<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Maintenance_and_Management&diff=1333Maintenance and Management2014-06-05T11:21:31Z<p>Bohme001: /* Maintenance and Management */</p>
<hr />
<div>== Maintenance and Management ==<br />
<br />
As of April 2014, questions should be directed to the Service Desk IT.<br />
<br />
This can be done via the mail: servicedesk.it@wur.nl <br />
And it can be done via the telephone: +31 317 488888<br />
Please give your name and phonenumber and tell that your mail/call is about the HPC for Agrogenomics and give the company you are working for.<br />
When you call the servicedesk, give also your email address.<br />
<br />
<br />
== Maintenance June 11'th 2014 ==<br />
There will be mainly firmware maintenance between 8h and 13h CET . Because network controller and storage controller firmware will be upgrades, all servers need to be rebooted and also will have network hick ups. So to prevent job and/or data corruption, the HPC will be shut down during this maintenance window. Running jobs will be killed!</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1320Using Slurm2014-04-03T14:07:55Z<p>Bohme001: /* Batch script */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br><br />
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br><br />
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Low queue jobs.<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_High up infinite 12 down* node[043-048,055-060]<br />
ABGC_High up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_High up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Std up infinite 12 down* node[043-048,055-060]<br />
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Std up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Low up infinite 12 down* node[043-048,055-060]<br />
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Low up infinite 44 idle node[001,006-042,049-054]<br />
</source><br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC_Std<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=ABGC_Std<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state when there are not enough resources available to this job.<br />
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1319Using Slurm2014-04-03T14:06:33Z<p>Bohme001: /* Queues */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 3 queues (in slurm called partitions) : a high, a standard and a low priority queue.<br><br />
The High queue provides the highest priority to jobs (20) then the standard queue (10). In the low priority queue (0)<br><br />
jobs will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Low queue jobs.<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_High up infinite 12 down* node[043-048,055-060]<br />
ABGC_High up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_High up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Std up infinite 12 down* node[043-048,055-060]<br />
ABGC_Std up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Std up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Low up infinite 12 down* node[043-048,055-060]<br />
ABGC_Low up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Low up infinite 44 idle node[001,006-042,049-054]<br />
</source><br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state when there are not enough resources available to this job.<br />
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=User:Bohme001&diff=1318User:Bohme0012014-04-02T11:07:50Z<p>Bohme001: </p>
<hr />
<div>Linux system administrator at FB-IT Infrastructure<br><br />
Our team is maintaining +200 Linux servers running mainly on RedHat Enterprise Server and some specials on Ubuntu Server.<br><br />
All HPC AgroGenomics hosts are running Scientific Linux version 6</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=User:Bohme001&diff=1317User:Bohme0012014-04-02T11:05:27Z<p>Bohme001: Created page with "Linux system administrator at FB-IT Infrastructure Our team is maintaining +200 Linux servers running mainly on RedHat Enterprise Server and some specials on Ubuntu Server. Al..."</p>
<hr />
<div>Linux system administrator at FB-IT Infrastructure<br />
Our team is maintaining +200 Linux servers running mainly on RedHat Enterprise Server and some specials on Ubuntu Server.<br />
All HPC AgroGenomics hosts are running Scientific Linux version 6</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=MPI_on_B4F_cluster&diff=1305MPI on B4F cluster2014-03-28T13:10:45Z<p>Bohme001: /* A mvapich2 sbatch example */</p>
<hr />
<div><br />
== A simple 'Hello World' example ==<br />
Consider the following simple MPI version, in C, of the 'Hello World' example:<br />
<br />
<source lang='cpp'><br />
#include <stdio.h><br />
#include <mpi.h><br />
int main(int argc, char ** argv) {<br />
int size,rank,namelen;<br />
char processor_name[MPI_MAX_PROCESSOR_NAME];<br />
MPI_Init(&argc, &argv);<br />
MPI_Comm_rank(MPI_COMM_WORLD,&rank);<br />
MPI_Comm_size(MPI_COMM_WORLD,&size);<br />
MPI_Get_processor_name(processor_name, &namelen);<br />
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);<br />
MPI_Finalize();<br />
}<br />
</source><br />
<br />
Before compiling, make sure that the compilers that are required available.<br />
<source lang='bash'><br />
module list<br />
</source><br />
<br />
To avoid conflicts between libraries, the safest way is purging all modules:<br />
<source lang='bash'><br />
module purge<br />
</source><br />
<br />
The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.<br />
<source lang='bash'><br />
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7<br />
</source><br />
<br />
Compile the <code>hello_mpi.c</code> code.<br />
<source lang='bash'><br />
mpicc hello_mpi.c -o test_hello_world<br />
</source><br />
<br />
If desired, a list of libraries compiled into the executable can be viewed:<br />
<source lang='bash'><br />
ldd test_hello_world<br />
</source><br />
<br />
linux-vdso.so.1 => (0x00002aaaaaacb000)<br />
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)<br />
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)<br />
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)<br />
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)<br />
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)<br />
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)<br />
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)<br />
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)<br />
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)<br />
<br />
Running the executable on two nodes, with four tasks per node, can be done like this:<br />
<source lang='bash'><br />
srun --nodes=2 --ntasks-per-node=4 --partition=ABGC --mpi=openmpi ./test_hello_world<br />
</source><br />
<br />
This will result in the following output:<br />
Hello MPI! Process 4 of 8 on node011<br />
Hello MPI! Process 1 of 8 on node010<br />
Hello MPI! Process 7 of 8 on node011<br />
Hello MPI! Process 6 of 8 on node011<br />
Hello MPI! Process 5 of 8 on node011<br />
Hello MPI! Process 2 of 8 on node010<br />
Hello MPI! Process 0 of 8 on node010<br />
Hello MPI! Process 3 of 8 on node010<br />
<br />
== A mvapich2 sbatch example ==<br />
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.<br />
<source lang='bash'><br />
$ module load mvapich2/gcc<br />
$ vim batch.sh<br />
#!/bin/sh<br />
#SBATCH --account=projectx<br />
#SBATCH --time=0<br />
#SBATCH -n 32<br />
#SBATCH --constraint=normalmem<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=MPItest<br />
#SBATCH --partition=ABGC_Production<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=user@wur.nl<br />
<br />
echo "Starting at `date`"<br />
echo "Running on hosts: $SLURM_NODELIST"<br />
echo "Running on $SLURM_NNODES nodes."<br />
echo "Running on $SLURM_NPROCS processors."<br />
echo "Current working directory is `pwd`"<br />
# echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"<br />
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0<br />
<br />
mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10 <br />
<br />
echo "Program finished with exit code $? at: `date`"<br />
<br />
$ sbatch batch.sh<br />
<br />
</source></div>Bohme001https://wiki.anunna.wur.nl/index.php?title=MPI_on_B4F_cluster&diff=1304MPI on B4F cluster2014-03-28T13:10:18Z<p>Bohme001: /* A mvapich2 sbatch example */</p>
<hr />
<div><br />
== A simple 'Hello World' example ==<br />
Consider the following simple MPI version, in C, of the 'Hello World' example:<br />
<br />
<source lang='cpp'><br />
#include <stdio.h><br />
#include <mpi.h><br />
int main(int argc, char ** argv) {<br />
int size,rank,namelen;<br />
char processor_name[MPI_MAX_PROCESSOR_NAME];<br />
MPI_Init(&argc, &argv);<br />
MPI_Comm_rank(MPI_COMM_WORLD,&rank);<br />
MPI_Comm_size(MPI_COMM_WORLD,&size);<br />
MPI_Get_processor_name(processor_name, &namelen);<br />
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);<br />
MPI_Finalize();<br />
}<br />
</source><br />
<br />
Before compiling, make sure that the compilers that are required available.<br />
<source lang='bash'><br />
module list<br />
</source><br />
<br />
To avoid conflicts between libraries, the safest way is purging all modules:<br />
<source lang='bash'><br />
module purge<br />
</source><br />
<br />
The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.<br />
<source lang='bash'><br />
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7<br />
</source><br />
<br />
Compile the <code>hello_mpi.c</code> code.<br />
<source lang='bash'><br />
mpicc hello_mpi.c -o test_hello_world<br />
</source><br />
<br />
If desired, a list of libraries compiled into the executable can be viewed:<br />
<source lang='bash'><br />
ldd test_hello_world<br />
</source><br />
<br />
linux-vdso.so.1 => (0x00002aaaaaacb000)<br />
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)<br />
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)<br />
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)<br />
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)<br />
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)<br />
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)<br />
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)<br />
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)<br />
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)<br />
<br />
Running the executable on two nodes, with four tasks per node, can be done like this:<br />
<source lang='bash'><br />
srun --nodes=2 --ntasks-per-node=4 --partition=ABGC --mpi=openmpi ./test_hello_world<br />
</source><br />
<br />
This will result in the following output:<br />
Hello MPI! Process 4 of 8 on node011<br />
Hello MPI! Process 1 of 8 on node010<br />
Hello MPI! Process 7 of 8 on node011<br />
Hello MPI! Process 6 of 8 on node011<br />
Hello MPI! Process 5 of 8 on node011<br />
Hello MPI! Process 2 of 8 on node010<br />
Hello MPI! Process 0 of 8 on node010<br />
Hello MPI! Process 3 of 8 on node010<br />
<br />
== A mvapich2 sbatch example ==<br />
A mpi job using mvapich2 on 32 cores, using the normal compute nodes and the fast infiniband interconnect for RDMA traffic.<br />
<source lang='bash'><br />
$ module load mvapich2/gcc<br />
$ vim batch.sh<br />
#!/bin/sh<br />
#SBATCH --account=projectx<br />
#SBATCH --time=0<br />
#SBATCH -n 32<br />
#SBATCH --constraint=normalmem<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=MPItest<br />
#SBATCH --partition=ABGC_Production<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=user@wur.nl<br />
<br />
echo "Starting at `date`"<br />
echo "Running on hosts: $SLURM_NODELIST"<br />
echo "Running on $SLURM_NNODES nodes."<br />
echo "Running on $SLURM_NPROCS processors."<br />
echo "Current working directory is `pwd`"<br />
echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"<br />
<br />
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0<br />
<br />
mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10 <br />
<br />
echo "Program finished with exit code $? at: `date`"<br />
<br />
$ sbatch batch.sh<br />
<br />
</source></div>Bohme001https://wiki.anunna.wur.nl/index.php?title=MPI_on_B4F_cluster&diff=1303MPI on B4F cluster2014-03-28T13:08:02Z<p>Bohme001: /* A simple 'Hello World' example */</p>
<hr />
<div><br />
== A simple 'Hello World' example ==<br />
Consider the following simple MPI version, in C, of the 'Hello World' example:<br />
<br />
<source lang='cpp'><br />
#include <stdio.h><br />
#include <mpi.h><br />
int main(int argc, char ** argv) {<br />
int size,rank,namelen;<br />
char processor_name[MPI_MAX_PROCESSOR_NAME];<br />
MPI_Init(&argc, &argv);<br />
MPI_Comm_rank(MPI_COMM_WORLD,&rank);<br />
MPI_Comm_size(MPI_COMM_WORLD,&size);<br />
MPI_Get_processor_name(processor_name, &namelen);<br />
printf("Hello MPI! Process %d of %d on %s\n", rank, size, processor_name);<br />
MPI_Finalize();<br />
}<br />
</source><br />
<br />
Before compiling, make sure that the compilers that are required available.<br />
<source lang='bash'><br />
module list<br />
</source><br />
<br />
To avoid conflicts between libraries, the safest way is purging all modules:<br />
<source lang='bash'><br />
module purge<br />
</source><br />
<br />
The load both gcc and openmpi libraries. If modules were purged, then slurm needs to be reloaded too.<br />
<source lang='bash'><br />
module load gcc/4.8.1 openmpi/gcc/64/1.6.5 slurm/2.5.7<br />
</source><br />
<br />
Compile the <code>hello_mpi.c</code> code.<br />
<source lang='bash'><br />
mpicc hello_mpi.c -o test_hello_world<br />
</source><br />
<br />
If desired, a list of libraries compiled into the executable can be viewed:<br />
<source lang='bash'><br />
ldd test_hello_world<br />
</source><br />
<br />
linux-vdso.so.1 => (0x00002aaaaaacb000)<br />
libmpi.so.1 => /cm/shared/apps/openmpi/gcc/64/1.6.5/lib64/libmpi.so.1 (0x00002aaaaaccd000)<br />
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab080000)<br />
libm.so.6 => /lib64/libm.so.6 (0x00002aaaab284000)<br />
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x0000003e29400000)<br />
librt.so.1 => /lib64/librt.so.1 (0x00002aaaab509000)<br />
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab711000)<br />
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaab92a000)<br />
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabb2e000)<br />
libc.so.6 => /lib64/libc.so.6 (0x00002aaaabd4b000)<br />
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)<br />
<br />
Running the executable on two nodes, with four tasks per node, can be done like this:<br />
<source lang='bash'><br />
srun --nodes=2 --ntasks-per-node=4 --partition=ABGC --mpi=openmpi ./test_hello_world<br />
</source><br />
<br />
This will result in the following output:<br />
Hello MPI! Process 4 of 8 on node011<br />
Hello MPI! Process 1 of 8 on node010<br />
Hello MPI! Process 7 of 8 on node011<br />
Hello MPI! Process 6 of 8 on node011<br />
Hello MPI! Process 5 of 8 on node011<br />
Hello MPI! Process 2 of 8 on node010<br />
Hello MPI! Process 0 of 8 on node010<br />
Hello MPI! Process 3 of 8 on node010<br />
<br />
== A mvapich2 sbatch example ==<br />
A mpi job using mvapich2 on 32 cores using the fast infiniband interconnect for RDMA traffic.<br />
<source lang='bash'><br />
$ module load mvapich2/gcc<br />
$ vim batch.sh<br />
#!/bin/sh<br />
#SBATCH --account=projectx<br />
#SBATCH --time=0<br />
#SBATCH -n 32<br />
#SBATCH --constraint=normalmem<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=MPItest<br />
#SBATCH --partition=ABGC_Production<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=user@wur.nl<br />
<br />
echo "Starting at `date`"<br />
echo "Running on hosts: $SLURM_NODELIST"<br />
echo "Running on $SLURM_NNODES nodes."<br />
echo "Running on $SLURM_NPROCS processors."<br />
echo "Current working directory is `pwd`"<br />
echo "Env var MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE is $MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE"<br />
<br />
# export MPIR_CVAR_NEMESIS_TCP_NETWORK_IFACE=ib0<br />
<br />
mpirun -iface ib0 -np 32 ./tmf_par.out -NX 480 -NY 240 -alpha 11 -chi 1.3 -psi_b 5e-2 -beta 0.0 -zeta 3.5 -kT 0.10 <br />
<br />
echo "Program finished with exit code $? at: `date`"<br />
<br />
</source></div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1137Using Slurm2014-02-21T14:47:39Z<p>Bohme001: /* Check on a pending job */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state when there are not enough resources available to this job.<br />
In this example I sumbit a job, check the status and after finding out is it '''pending''' I'll check when is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
So it seems this job will problably start the next day, but's thats no guarantee it will start indeed.<br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1136Using Slurm2014-02-21T14:45:22Z<p>Bohme001: /* Query a specific active job: scontrol */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
=== Check on a pending job ===<br />
A submitted job could result in a pending state of tje job when there are not enough resources available to this job.<br />
In this example I sumbit a job, check it status and after finding out is it pending I'll check whan is probably will start.<br />
<source lang='bash'><br />
[@nfs01 jobs]$ sbatch hpl_student.job<br />
Submitted batch job 740338<br />
<br />
[@nfs01 jobs]$ squeue -l -j 740338<br />
Fri Feb 21 15:32:31 2014<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PENDING 0:00 1-00:00:00 1 (ReqNodeNotAvail)<br />
<br />
[@nfs01 jobs]$ squeue --start -j 740338<br />
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)<br />
740338 ABGC_Stud HPLstude bohme999 PD 2014-02-22T15:31:48 1 (ReqNodeNotAvail)<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1135Using Slurm2014-02-21T14:41:01Z<p>Bohme001: /* Monitoring submitted and quering jobs */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1134Using Slurm2014-02-21T14:40:33Z<p>Bohme001: /* Query a specific active job: scontrol */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted and quering jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
=== Query a specific active job: scontrol ===<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1133Using Slurm2014-02-21T14:40:02Z<p>Bohme001: /* monitoring submitted jobs: squeue */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== Monitoring submitted and quering jobs ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1132Using Slurm2014-02-21T14:37:49Z<p>Bohme001: /* Queues */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
=== monitoring submitted jobs: squeue ===<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1131Using Slurm2014-02-21T14:37:27Z<p>Bohme001: /* Queues */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
== Queues ==<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
=== monitoring submitted jobs: squeue ===<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1130Using Slurm2014-02-21T14:34:35Z<p>Bohme001: /* monitoring submitted jobs: squeue */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
=== monitoring submitted jobs: squeue ===<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1129Using Slurm2014-02-21T13:55:45Z<p>Bohme001: /* = Interactive X11/GUI jobs */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ===<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1128Using Slurm2014-02-21T13:55:20Z<p>Bohme001: /* Submitting multiple jobs */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<br />
When requesting multiple tasks, you may or may not want the job to be partitioned among multiple nodes. You can specify the minimum number of nodes using the <code>-N</code> or <code>--node</code> flag. If you provide only one number, this will be minimum and maximum at the same time. For instance:<br />
<source lang='bash'><br />
#SBATCH --nodes=1<br />
</source><br />
This should force your job to be scheduled to a single node.<br />
<br />
Because the cluster has a hybrid configuration, i.e. normal and fat nodes, it may be prudent to schedule your job specifically for one or the other node type, depending for instance on memory requirements. This can be done by using the <code>-C</code> or <code>--constraints</code> flag.<br />
<source lang='bash'><br />
#SBATCH --constraint=normalmem<br />
</source><br />
The example above will result in jobs being scheduled to the regular compute nodes. By using <code>largemem</code> as option the job will specifically be scheduled to one of the fat nodes. <br />
<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
=== Interactive X11/GUI jobs ==<br />
Slurm will forward your X11 credentials to the first (or even all) node for a job with the (undocumented) --x11 option.<br />
For example, an interactive session for 1 hour with HPL using eigth cores can be started with:<br />
<source lang='bash'>module load hpl/2.1<br />
srun --ntasks=1 --cpus-per-task=8 --time=1:00:00 --pty --x11=first xhpl<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1126Using Slurm2014-01-16T08:25:03Z<p>Bohme001: /* Get overview of past and current jobs: sacct */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code!!State!!Description<br />
|-<br />
|CA ||CANCELLED|| Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
|-<br />
|CD|| COMPLETED|| Job has terminated all processes on all nodes.<br />
|-<br />
|CF|| CONFIGURING|| Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
|-<br />
|CG|| COMPLETING|| Job is in the process of completing. Some processes on some nodes may still be active.<br />
|-<br />
|F|| FAILED|| Job terminated with non-zero exit code or other failure condition.<br />
|-<br />
|NF|| NODE_FAIL|| Job terminated due to failure of one or more allocated nodes.<br />
|-<br />
|PD|| PENDING|| Job is awaiting resource allocation.<br />
|-<br />
|R|| RUNNING|| Job currently has an allocation.<br />
|-<br />
|S|| SUSPENDED|| Job has an allocation, but execution has been suspended.<br />
|-<br />
|TO|| TIMEOUT|| Job terminated upon reaching its time limit.<br />
|-<br />
|-<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1125Using Slurm2014-01-16T08:21:46Z<p>Bohme001: /* Get overview of past and current jobs: sacct */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
'''Job Status Codes'''<br />
<br />
Typically your job will be either in the Running state of PenDing state. However here is a breakdown of all the states that your job could be in.<br />
<br />
{| class="wikitable"<br />
|-<br />
!Code <br />
!State <br />
!Description<br />
|-<br />
CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.<br />
CD COMPLETED Job has terminated all processes on all nodes.<br />
CF CONFIGURING Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).<br />
CG COMPLETING Job is in the process of completing. Some processes on some nodes may still be active.<br />
F FAILED Job terminated with non-zero exit code or other failure condition.<br />
NF NODE_FAIL Job terminated due to failure of one or more allocated nodes.<br />
PD PENDING Job is awaiting resource allocation.<br />
R RUNNING Job currently has an allocation.<br />
S SUSPENDED Job has an allocation, but execution has been suspended.<br />
TO TIMEOUT Job terminated upon reaching its time limit.<br />
|}<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Setting_TMPDIR&diff=1080Setting TMPDIR2013-12-29T11:03:24Z<p>Bohme001: </p>
<hr />
<div>'''Note: all nodes are partioned with a large /tmp (+- 400G).'''<br />
<br />
Many programs require writing intermediary or temporary information. A process may even require writing to a temporary location without a user knowing it. For instance, the command <code>sort</code>, part of the <code>bash</code> toolkit, requires a lot of temporary file space when sorting large volumes of data. Often programs take the system default, which is usually <code>/tmp</code> on Linux systems. The <code>/tmp</code> partition is however often too limited in size and can get filled up. When this happens all users that require write access to <code>/tmp</code> will experience problems in running jobs, which can range from unexpected quitting of processes, to erroneous output. <br />
<br />
== Overriding the system default - works for many applications (but not all) ==<br />
<br />
Users can set a custom temporary folder location by setting the <code>TMPDIR</code> environment variable. A custom temporary folder location is best placed in the 'scratch' space on the /lustre filesystem, because this will ensure periodic tidying up of the custom temporary directory thereby reducing the opportunity for very large temporary files that have gone unnoticed to remain on the filesystem. In addition, this is in line with protocols for the default system temporary directory <code>/tmp</code> as this directory is also periodically purged of older files.<br />
<br />
First, create a temporary directory on user designated space on on the 'scratch' partition:<br />
<source lang='bash'><br />
mkdir /lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
Set the <code>TMPDIR</code> variable to point at the custom directory in either <code>~/.bash_profile</code> or <code>~/.bashrc</code> using a command-line editor of choice (vi, emacs, nano, etc), adding the following line of code:<br />
<source lang='bash'><br />
export TMPDIR=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
The temporary directory will be set to the custom location once you log in. For the current open terminal to set the new location immediately, without need to log out and in again:<br />
<source lang='bash'><br />
source ~/.bash_profile<br />
</source><br />
<br />
Note that certain (compiled) apps may in fact ignore the <code>TMPDIR</code> environment variable. These can include Java-apps and c++/g++ compiled apps.<br />
<br />
== Setting custom temporary directory for Java applications ==<br />
<br />
Java applications often do not respond properly to re-setting of the system default by manipulating the <code>TMPDIR</code> environment variable. Java behaviour can be modified by setting the <code>_JAVA_OPTIONS</code> environment variable:<br />
<source lang='bash'><br />
export _JAVA_OPTIONS=-Djava.io.tmpdir=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
Note that this does not - strictly speaking - manipulates a temporary directory specific environment variable; rather it sets a Java specific environment variable that allows to pass options to Java on runtime.<br />
<br />
== See also ==<br />
[[Main_Page#Being_in_control_of_Environment_parameters | Controling environment on the Agrogenomics cluster]]<br />
<br />
== External links ==</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Setting_TMPDIR&diff=1079Setting TMPDIR2013-12-29T11:03:01Z<p>Bohme001: </p>
<hr />
<div>'''Note: all nodes are partioned with a large /tmp (+- 400G), /local is much smaller!'''<br />
<br />
Many programs require writing intermediary or temporary information. A process may even require writing to a temporary location without a user knowing it. For instance, the command <code>sort</code>, part of the <code>bash</code> toolkit, requires a lot of temporary file space when sorting large volumes of data. Often programs take the system default, which is usually <code>/tmp</code> on Linux systems. The <code>/tmp</code> partition is however often too limited in size and can get filled up. When this happens all users that require write access to <code>/tmp</code> will experience problems in running jobs, which can range from unexpected quitting of processes, to erroneous output. <br />
<br />
== Overriding the system default - works for many applications (but not all) ==<br />
<br />
Users can set a custom temporary folder location by setting the <code>TMPDIR</code> environment variable. A custom temporary folder location is best placed in the 'scratch' space on the /lustre filesystem, because this will ensure periodic tidying up of the custom temporary directory thereby reducing the opportunity for very large temporary files that have gone unnoticed to remain on the filesystem. In addition, this is in line with protocols for the default system temporary directory <code>/tmp</code> as this directory is also periodically purged of older files.<br />
<br />
First, create a temporary directory on user designated space on on the 'scratch' partition:<br />
<source lang='bash'><br />
mkdir /lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
Set the <code>TMPDIR</code> variable to point at the custom directory in either <code>~/.bash_profile</code> or <code>~/.bashrc</code> using a command-line editor of choice (vi, emacs, nano, etc), adding the following line of code:<br />
<source lang='bash'><br />
export TMPDIR=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
The temporary directory will be set to the custom location once you log in. For the current open terminal to set the new location immediately, without need to log out and in again:<br />
<source lang='bash'><br />
source ~/.bash_profile<br />
</source><br />
<br />
Note that certain (compiled) apps may in fact ignore the <code>TMPDIR</code> environment variable. These can include Java-apps and c++/g++ compiled apps.<br />
<br />
== Setting custom temporary directory for Java applications ==<br />
<br />
Java applications often do not respond properly to re-setting of the system default by manipulating the <code>TMPDIR</code> environment variable. Java behaviour can be modified by setting the <code>_JAVA_OPTIONS</code> environment variable:<br />
<source lang='bash'><br />
export _JAVA_OPTIONS=-Djava.io.tmpdir=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
Note that this does not - strictly speaking - manipulates a temporary directory specific environment variable; rather it sets a Java specific environment variable that allows to pass options to Java on runtime.<br />
<br />
== See also ==<br />
[[Main_Page#Being_in_control_of_Environment_parameters | Controling environment on the Agrogenomics cluster]]<br />
<br />
== External links ==</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1047Using Slurm2013-12-27T14:56:50Z<p>Bohme001: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
----<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1046Using Slurm2013-12-27T14:55:12Z<p>Bohme001: /* Defaults */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br><br />
'''The default run time for a job is 1 hour!''' <br><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1045Using Slurm2013-12-27T14:54:53Z<p>Bohme001: /* Defaults */</p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<cr><br />
'''The default run time for a job is 1 hour!''' <cr><br />
'''Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=1044Using Slurm2013-12-27T14:54:15Z<p>Bohme001: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<cr><br />
'''The default run time for a job is 1 hour!<br />
Default memory limit is 1024MB per node!'''<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --mem=2048<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<br />
<source lang='bash'><br />
#SBATCH --mem=2048<br />
</source><br />
SLURM imposes a memory limit on each job. By default, it is deliberately relatively small — 1024 MB per node. If your job uses more than that, you’ll get an error that your job Exceeded job memory limit. To set a larger limit, add to your job submission: <br />
<source lang='bash'><br />
#SBATCH --mem X<br />
</source><br />
<br />
where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:<br />
<source lang='bash'><br />
$ sacct -o MaxRSS -j JOBID<br />
</source><br />
where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit). If your job completed long in the past you may have to tell sacct to look further back in time by adding a start time with -S YYYY-MM-DD. Note that for parallel jobs spanning multiple nodes, this is the maximum memory used on any one node; if you’re not setting an even distribution of tasks per node (e.g. with –ntasks-per-node), the same job could have very different values when run at different times.<br />
<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=HPCwiki:About&diff=901HPCwiki:About2013-12-20T13:25:54Z<p>Bohme001: </p>
<hr />
<div>This Wiki is intended for members and affiliates of the HPC Agrogenomics of Wageningen UR only.</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=887Using Slurm2013-12-16T14:40:41Z<p>Bohme001: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
=== Defaults ===<br />
There is no default queue, so you need to specify which queue to use when submitting a job.<br />
The default run time for a job is 1 hour!<br />
<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Using_Slurm&diff=886Using Slurm2013-12-16T14:28:24Z<p>Bohme001: </p>
<hr />
<div>The resource allocation / scheduling software on the B4F Cluster is [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management SLURM]: '''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement.<br />
<br />
<br />
== Queues and defaults ==<br />
<br />
=== Queues ===<br />
Every organization has 2 queues (in slurm called partitions) : a production and a research queue. The production queue provides a higher priority to jobs (20) then the research queue (10).<br />
To find out which queues your account has been authorized for, type sinfo:<br />
<source lang='bash'><br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
ABGC_Production up infinite 12 down* node[043-048,055-060]<br />
ABGC_Production up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Production up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Research up infinite 12 down* node[043-048,055-060]<br />
ABGC_Research up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Research up infinite 44 idle node[001,006-042,049-054]<br />
ABGC_Student up infinite 12 down* node[043-048,055-060]<br />
ABGC_Student up infinite 6 mix fat[001-002],node[002-005]<br />
ABGC_Student up infinite 44 idle node[001,006-042,049-054]<br />
</source> <br />
WUR organizations also do have a Student queue. Jobs in this queue will be resubmitted if a job with higer priority needs cluster resources and those resoruces are accupied by a Student queue jobs.<br />
<br />
== Submitting jobs: sbatch ==<br />
<br />
=== Example ===<br />
Consider this simple python3 script that should calculate Pi to 1 million digits:<br />
<source lang='python'><br />
from decimal import *<br />
D=Decimal<br />
getcontext().prec=10000000<br />
p=sum(D(1)/16**k*(D(4)/(8*k+1)-D(2)/(8*k+4)-D(1)/(8*k+5)-D(1)/(8*k+6))for k in range(411))<br />
print(str(p)[:10000002])<br />
</source> <br />
<br />
=== Loading modules ===<br />
In order for this script to run, the first thing that is needed is that Python3, which is not the default Python version on the cluster, is load into your environment. Availability of (different versions of) software can be checked by the following command:<br />
module avail<br />
<br />
In the list you should note that python3 is indeed available to be loaded, which then can be loaded with the following command:<br />
module load python/3.3.3<br />
<br />
=== Batch script ===<br />
The following shell/slurm script can then be used to schedule the job using the sbatch command:<br />
<source lang='bash'><br />
#!/bin/bash<br />
#SBATCH --account=773320000<br />
#SBATCH --time=1200<br />
#SBATCH --ntasks=1<br />
#SBATCH --output=output_%j.txt<br />
#SBATCH --error=error_output_%j.txt<br />
#SBATCH --job-name=calc_pi.py<br />
#SBATCH --partition=ABGC<br />
#SBATCH --mail-type=ALL<br />
#SBATCH --mail-user=email@org.nl<br />
<br />
<br />
time python3 calc_pi.py<br />
</source><br />
Explanation of used SBATCH parameters:<br />
<source lang='bash'><br />
#SBATCH --account=773320000<br />
</source><br />
Charge resources used by this job to specified account. The account is an arbitrary string. The account name may be changed after job submission using the scontrol command. For WUR users a projectnumber or KTP number would be advisable.<br />
<source lang='bash'><br />
#SBATCH --time=1200<br />
</source><br />
A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". So in this example the job will run for a maximum of 1200 minutes.<br />
<source lang='bash'><br />
#SBATCH --ntasks=1<br />
</source><br />
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the SLURM controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.<br />
<source lang='bash'><br />
#SBATCH --output=output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard output directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --error=error_output_%j.txt<br />
</source><br />
Instruct SLURM to connect the batch script's standard error directly to the file name specified in the "filename pattern". By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the --input option for filename specification options.<br />
<source lang='bash'><br />
#SBATCH --job-name=calc_pi.py<br />
</source><br />
Specify a name for the job allocation. The specified name will appear along with the job id number when querying running jobs on the system. The default is the name of the batch script, or just "sbatch" if the script is read on sbatch's standard input.<br />
<source lang='bash'><br />
#SBATCH --partition=research<br />
</source><br />
Request a specific partition for the resource allocation. It is prefered to use your organizations partition.<br />
<source lang='bash'><br />
#SBATCH --mail-type=ALL<br />
</source><br />
Notify user by email when certain event types occur. Valid type values are BEGIN, END, FAIL, REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.<br />
<source lang='bash'><br />
#SBATCH --mail-user=email@org.nl<br />
</source><br />
Email address to use.<br />
<br />
=== Submitting ===<br />
The script, assuming it was named 'run_calc_pi.sh', can then be posted using the following command:<br />
<source lang='bash'><br />
sbatch run_calc_pi.sh<br />
</source><br />
<br />
=== Submitting multiple jobs ===<br />
Assuming there are 10 job scripts, name runscript_1.sh through runscript_10.sh, all these scripts can be submitted using the following line of shell code:<br />
<source lang='bash'>for i in `seq 1 10`; do echo $i; sbatch runscript_$i.sh;done<br />
</source><br />
<br />
== monitoring submitted jobs: squeue ==<br />
Once a job is submitted, the status can be monitored using the <code>squeue</code> command. The <code>squeue</code> command has a number of parameters for monitoring specific properties of the jobs such as time limit.<br />
<br />
=== Generic monitoring of all running jobs ===<br />
<source lang='bash'><br />
squeue<br />
</source><br />
<br />
You should then get a list of jobs that are running at that time on the cluster, for the example on how to submit using the 'sbatch' command, it may look like so:<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
3396 ABGC BOV-WUR- megen002 R 27:26 1 node004<br />
3397 ABGC BOV-WUR- megen002 R 27:26 1 node005<br />
3398 ABGC BOV-WUR- megen002 R 27:26 1 node006<br />
3399 ABGC BOV-WUR- megen002 R 27:26 1 node007<br />
3400 ABGC BOV-WUR- megen002 R 27:26 1 node008<br />
3401 ABGC BOV-WUR- megen002 R 27:26 1 node009<br />
3385 research BOV-WUR- megen002 R 44:38 1 node049<br />
3386 research BOV-WUR- megen002 R 44:38 1 node050<br />
3387 research BOV-WUR- megen002 R 44:38 1 node051<br />
3388 research BOV-WUR- megen002 R 44:38 1 node052<br />
3389 research BOV-WUR- megen002 R 44:38 1 node053<br />
3390 research BOV-WUR- megen002 R 44:38 1 node054<br />
3391 research BOV-WUR- megen002 R 44:38 3 node[049-051]<br />
3392 research BOV-WUR- megen002 R 44:38 3 node[052-054]<br />
3393 research BOV-WUR- megen002 R 44:38 1 node001<br />
3394 research BOV-WUR- megen002 R 44:38 1 node002<br />
3395 research BOV-WUR- megen002 R 44:38 1 node003<br />
<br />
=== Monitoring time limit set for a specific job ===<br />
The default time limit is set at one hour. Estimated run times need to be specified when running jobs. To see what the time limit is that is set for a certain job, this can be done using the <code>squeue</code> command.<br />
<source lang='bash'><br />
squeue -l -j 3532<br />
</source><br />
Information similar to the following should appear:<br />
Fri Nov 29 15:41:00 2013<br />
JOBID PARTITION NAME USER STATE TIME TIMELIMIT NODES NODELIST(REASON)<br />
3532 ABGC BOV-WUR- megen002 RUNNING 2:47:03 3-08:00:00 1 node054<br />
<br />
== Query a specific active job: scontrol ==<br />
Show all the details of a currently active job, so not a completed job.<br />
<source lang='bash'><br />
nfs01 ~]$ scontrol show jobid 4241<br />
JobId=4241 Name=WB20F06<br />
UserId=megen002(16795409) GroupId=domain users(16777729)<br />
Priority=1 Account=(null) QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0<br />
RunTime=02:55:25 TimeLimit=3-08:00:00 TimeMin=N/A<br />
SubmitTime=2013-12-09T13:37:29 EligibleTime=2013-12-09T13:37:29<br />
StartTime=2013-12-09T13:37:29 EndTime=2013-12-12T21:37:29<br />
PreemptTime=None SuspendTime=None SecsPreSuspend=0<br />
Partition=research AllocNode:Sid=nfs01:21799<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=node023<br />
BatchHost=node023<br />
NumNodes=1 NumCPUs=4 CPUs/Task=1 ReqS:C:T=*:*:*<br />
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br />
Features=(null) Gres=(null) Reservation=(null)<br />
Shared=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/lustre/scratch/WUR/ABGC/...<br />
WorkDir=/lustre/scratch/WUR/ABGC/...<br />
</source><br />
<br />
== Removing jobs from a list: scancel ==<br />
If for some reason you want to delete a job that is either in the queue or already running, you can remove it using the 'scancel' command. The 'scancel' command takes the jobid as a parameter. For the example above, this would be done using the following code:<br />
<source lang='bash'><br />
scancel 3401<br />
</source><br />
<br />
== Allocating resources interactively: sallocate ==<br />
< text here><br />
<br />
== Get overview of past and current jobs: sacct ==<br />
To do some accounting on past and present jobs, and to see whether they ran to completion, you can do:<br />
<source lang='bash'><br />
sacct<br />
</source><br />
This should provide information similar to the following:<br />
<br />
JobID JobName Partition Account AllocCPUS State ExitCode <br />
------------ ---------- ---------- ---------- ---------- ---------- -------- <br />
3385 BOV-WUR-58 research 12 COMPLETED 0:0 <br />
3385.batch batch 1 COMPLETED 0:0 <br />
3386 BOV-WUR-59 research 12 CANCELLED+ 0:0 <br />
3386.batch batch 1 CANCELLED 0:15 <br />
3528 BOV-WUR-59 ABGC 16 RUNNING 0:0 <br />
3529 BOV-WUR-60 ABGC 16 RUNNING 0:0<br />
<br />
Or in more detail for a specific job:<br />
<source lang='bash'><br />
sacct --format=jobid,jobname,account,partition,ntasks,alloccpus,elapsed,state,exitcode -j 4220<br />
</source><br />
This should provide information about job id 4220:<br />
<br />
JobID JobName Account Partition NTasks AllocCPUS Elapsed State ExitCode <br />
------------ ---------- ---------- ---------- -------- ---------- ---------- ---------- -------- <br />
4220 PreProces+ research 3 00:30:52 COMPLETED 0:0 <br />
4220.batch batch 1 1 00:30:52 COMPLETED 0:0<br />
<br />
== Running MPI jobs on B4F cluster ==<br />
<br />
[[MPI_on_B4F_cluster | Main article: MPI on B4F Cluster]]<br />
< text here ><br />
<br />
== Understanding which resources are available to you: sinfo ==<br />
By using the 'sinfo' command you can retrieve information on which 'Partitions' are available to you. A 'Partition' using SLURM is similar to the 'queue' when submitting using the Sun Grid Engine ('qsub'). The different Partitions grant different levels of resource allocation. Not all defined Partitions will be available to any given person. E.g., Master students will only have the Student Partition available, researchers at the ABGC will have 'student', 'research', and 'ABGC' partitions available. The higher the level of resource allocation, though, the higher the cost per compute-hour. The default Partition is the 'student' partition. A full list of Partitions can be found from the Bright Cluster Manager webpage.<br />
<br />
<source lang='bash'><br />
sinfo<br />
</source><br />
<br />
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br />
student* up infinite 12 down* node[043-048,055-060]<br />
student* up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
research up infinite 12 down* node[043-048,055-060]<br />
research up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
ABGC up infinite 12 down* node[043-048,055-060]<br />
ABGC up infinite 50 idle fat[001-002],node[001-042,049-054]<br />
<br />
== See also ==<br />
* [[B4F_cluster | B4F Cluster]]<br />
* [[BCM_on_B4F_cluster | BCM on B4F cluster]]<br />
* [[SLURM_Compare | SLURM compared to other common schedulers]]<br />
* [[Setting_up_Python_virtualenv | Setting up and using a virtual environment for Python3 ]]<br />
<br />
== External links ==<br />
* [http://slurm.schedmd.com Slurm official documentation]<br />
* [http://en.wikipedia.org/wiki/Simple_Linux_Utility_for_Resource_Management Slurm on Wikipedia]<br />
* [http://www.youtube.com/watch?v=axWffyrk3aY Slurm Tutorial on Youtube]</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=File:BCM.png&diff=885File:BCM.png2013-12-16T13:52:19Z<p>Bohme001: Bohme001 uploaded a new version of &quot;File:BCM.png&quot;</p>
<hr />
<div>BCM portal</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Setting_TMPDIR&diff=884Setting TMPDIR2013-12-13T14:51:35Z<p>Bohme001: </p>
<hr />
<div>'''Note: all nodes are partioned with a large /local (+- 400G), /tmp is much smaller!'''<br />
<br />
Many programs require writing intermediary or temporary information. A process may even require writing to a temporary location without a user knowing it. For instance, the command <code>sort</code>, part of the <code>bash</code> toolkit, requires a lot of temporary file space when sorting large volumes of data. Often programs take the system default, which is usually <code>/tmp</code> on Linux systems. The <code>/tmp</code> partition is however often too limited in size and can get filled up. When this happens all users that require write access to <code>/tmp</code> will experience problems in running jobs, which can range from unexpected quitting of processes, to erroneous output. <br />
<br />
Users can set a custom temporary folder location by setting the <code>TMPDIR</code> environment variable. A custom temporary folder location is best placed in the 'scratch' space on the /lustre filesystem, because this will ensure periodic tidying up of the custom temporary directory thereby reducing the opportunity for very large temporary files that have gone unnoticed to remain on the filesystem. In addition, this is in line with protocols for the default system temporary directory <code>/tmp</code> as this directory is also periodically purged of older files.<br />
<br />
First, create a temporary directory on user designated space on on the 'scratch' partition:<br />
<source lang='bash'><br />
mkdir /lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
Set the <code>TMPDIR</code> variable to point at the custom directory in either <code>~/.bash_profile</code> or <code>~/.bashrc</code> using a command-line editor of choice (vi, emacs, nano, etc), adding the following line of code:<br />
<source lang='bash'><br />
export TMPDIR=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
The temporary directory will be set to the custom location once you log in. For the current open terminal to set the new location immediately, without need to log out and in again:<br />
<source lang='bash'><br />
source ~/.bash_profile<br />
</source></div>Bohme001https://wiki.anunna.wur.nl/index.php?title=Setting_TMPDIR&diff=883Setting TMPDIR2013-12-13T14:51:05Z<p>Bohme001: </p>
<hr />
<div>Many programs require writing intermediary or temporary information. A process may even require writing to a temporary location without a user knowing it. For instance, the command <code>sort</code>, part of the <code>bash</code> toolkit, requires a lot of temporary file space when sorting large volumes of data. Often programs take the system default, which is usually <code>/tmp</code> on Linux systems. The <code>/tmp</code> partition is however often too limited in size and can get filled up. When this happens all users that require write access to <code>/tmp</code> will experience problems in running jobs, which can range from unexpected quitting of processes, to erroneous output. <br />
<br />
'''Note: all nodes are partioned with a large /local (+- 400G), /tmp is much smaller!'''<br />
<br />
Users can set a custom temporary folder location by setting the <code>TMPDIR</code> environment variable. A custom temporary folder location is best placed in the 'scratch' space on the /lustre filesystem, because this will ensure periodic tidying up of the custom temporary directory thereby reducing the opportunity for very large temporary files that have gone unnoticed to remain on the filesystem. In addition, this is in line with protocols for the default system temporary directory <code>/tmp</code> as this directory is also periodically purged of older files.<br />
<br />
First, create a temporary directory on user designated space on on the 'scratch' partition:<br />
<source lang='bash'><br />
mkdir /lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
Set the <code>TMPDIR</code> variable to point at the custom directory in either <code>~/.bash_profile</code> or <code>~/.bashrc</code> using a command-line editor of choice (vi, emacs, nano, etc), adding the following line of code:<br />
<source lang='bash'><br />
export TMPDIR=/lustre/scratch/WUR/ABGC/[user]/tmp<br />
</source><br />
<br />
The temporary directory will be set to the custom location once you log in. For the current open terminal to set the new location immediately, without need to log out and in again:<br />
<source lang='bash'><br />
source ~/.bash_profile<br />
</source></div>Bohme001https://wiki.anunna.wur.nl/index.php?title=SLURM_Compare&diff=869SLURM Compare2013-12-10T11:56:57Z<p>Bohme001: /* Rosetta Stone of Workload Managers */</p>
<hr />
<div>=== Rosetta Stone of Workload Managers ===<br />
<br />
PBS/Torque, Slurm, LSF, SGE and LoadLeveler [http://slurm.schedmd.com/rosetta.html Rosetta Stone]<br />
<br />
This table lists the most common command, environment variables, and job specification options used by the major workload management systems: PBS/Torque, Slurm, LSF, SGE and LoadLeveler. Each of these workload managers has unique features, but the most commonly used functionality is available in all of these environments as listed in the table. This should be considered a work in progress and contributions to improve the document are welcome.<br />
{| class="wikitable"<br />
!User Commands!!PBS/Torque!!Slurm!!LSF!!SGE!!LoadLeveler<br />
|-<br />
||Job submission|| qsub [script_file]|| sbatch [script_file]|| bsub [script_file]|| qsub [script_file]|| llsubmit [script_file] <br />
|-<br />
||Job deletion ||qdel [job_id]|| scancel [job_id]|| bkill [job_id]|| qdel [job_id]|| llcancel [job_id] <br />
|-<br />
||Job status (by job)|| qstat [job_id]|| squeue [job_id]|| bjobs [job_id]|| qstat -u \* [-j job_id]|| llq -u [username] <br />
|-<br />
||Job status (by user)|| qstat -u [user_name]|| squeue -u [user_name]|| bjobs -u [user_name]|| qstat [-u user_name]|| llq -u [user_name] <br />
|-<br />
||Job hold ||qhold [job_id]|| scontrol hold [job_id]|| bstop [job_id]|| qhold [job_id]|| llhold -r [job_id] <br />
|-<br />
||Job release|| qrls [job_id]|| scontrol release [job_id]|| bresume [job_id]|| qrls [job_id]|| llhold -r [job_id] <br />
|-<br />
||Queue list|| qstat -Q|| squeue ||bqueues ||qconf -sql|| llclass <br />
|-<br />
||Node list ||pbsnodes -l|| sinfo -N OR scontrol show nodes|| bhosts|| qhost|| llstatus -L machine <br />
|-<br />
||Cluster status|| qstat -a|| sinfo|| bqueues|| qhost -q|| llstatus -L cluster <br />
|-<br />
||GUI|| xpbsmon|| sview|| xlsf OR xlsbatch|| qmon|| xload <br />
|-<br />
|-<br />
||'''Environment'''||'''PBS/Torque'''||'''Slurm'''||'''LSF'''||'''SGE'''||'''LoadLeveler''' <br />
|-<br />
||Job ID|| $PBS_JOBID|| $SLURM_JOBID|| $LSB_JOBID|| $JOB_ID|| $LOAD_STEP_ID <br />
|-<br />
||Submit Directory|| $PBS_O_WORKDIR|| $SLURM_SUBMIT_DIR|| $LSB_SUBCWD|| $SGE_O_WORKDIR|| $LOADL_STEP_INITDIR <br />
|-<br />
||Submit Host|| $PBS_O_HOST|| $SLURM_SUBMIT_HOST|| $LSB_SUB_HOST|| $SGE_O_HOST ||<br />
|-<br />
||Node List|| $PBS_NODEFILE|| $SLURM_JOB_NODELIST|| $LSB_HOSTS/LSB_MCPU_HOST|| $PE_HOSTFILE|| $LOADL_PROCESSOR_LIST <br />
|-<br />
||Job Array Index|| $PBS_ARRAYID|| $SLURM_ARRAY_TASK_ID|| $LSB_JOBINDEX|| $SGE_TASK_ID ||<br />
|-<br />
|-<br />
|-<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=SLURM_Compare&diff=868SLURM Compare2013-12-10T11:55:44Z<p>Bohme001: /* Rosetta Stone of Workload Managers */</p>
<hr />
<div>=== Rosetta Stone of Workload Managers ===<br />
<br />
PBS/Torque, Slurm, LSF, SGE and LoadLeveler [http://slurm.schedmd.com/rosetta.html Rosetta Stone]<br />
<br />
This table lists the most common command, environment variables, and job specification options used by the major workload management systems: PBS/Torque, Slurm, LSF, SGE and LoadLeveler. Each of these workload managers has unique features, but the most commonly used functionality is available in all of these environments as listed in the table. This should be considered a work in progress and contributions to improve the document are welcome.<br />
{| class="wikitable"<br />
!User Commands!!PBS/Torque!!Slurm!!LSF!!SGE!!LoadLeveler<br />
|-<br />
||Job submission|| qsub [script_file]|| sbatch [script_file]|| bsub [script_file]|| qsub [script_file]|| llsubmit [script_file] <br />
|-<br />
||Job deletion ||qdel [job_id]|| scancel [job_id]|| bkill [job_id]|| qdel [job_id]|| llcancel [job_id] <br />
|-<br />
||Job status (by job)|| qstat [job_id]|| squeue [job_id]|| bjobs [job_id]|| qstat -u \* [-j job_id]|| llq -u [username] <br />
|-<br />
||Job status (by user)|| qstat -u [user_name]|| squeue -u [user_name]|| bjobs -u [user_name]|| qstat [-u user_name]|| llq -u [user_name] <br />
|-<br />
||Job hold ||qhold [job_id]|| scontrol hold [job_id]|| bstop [job_id]|| qhold [job_id]|| llhold -r [job_id] <br />
|-<br />
||Job release|| qrls [job_id]|| scontrol release [job_id]|| bresume [job_id]|| qrls [job_id]|| llhold -r [job_id] <br />
|-<br />
||Queue list|| qstat -Q|| squeue ||bqueues ||qconf -sql|| llclass <br />
|-<br />
||Node list ||pbsnodes -l|| sinfo -N OR scontrol show nodes|| bhosts|| qhost|| llstatus -L machine <br />
|-<br />
||Cluster status|| qstat -a|| sinfo|| bqueues|| qhost -q|| llstatus -L cluster <br />
|-<br />
||GUI|| xpbsmon|| sview|| xlsf OR xlsbatch|| qmon|| xload <br />
|-<br />
|-<br />
||'''Environment'''||'''PBS/Torque'''||'''Slurm'''||'''LSF'''||'''SGE'''||'''LoadLeveler''' <br />
|-<br />
||Job ID|| $PBS_JOBID|| $SLURM_JOBID|| $LSB_JOBID|| $JOB_ID|| $LOAD_STEP_ID <br />
|-<br />
||Submit Directory|| $PBS_O_WORKDIR|| $SLURM_SUBMIT_DIR|| $LSB_SUBCWD|| $SGE_O_WORKDIR|| $LOADL_STEP_INITDIR <br />
|-<br />
||Submit Host|| $PBS_O_HOST|| $SLURM_SUBMIT_HOST|| $LSB_SUB_HOST|| $SGE_O_HOST ||<br />
|-<br />
||Node List|| $PBS_NODEFILE|| $SLURM_JOB_NODELIST|| $LSB_HOSTS/LSB_MCPU_HOST|| $PE_HOSTFILE|| $LOADL_PROCESSOR_LIST <br />
|-<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=SLURM_Compare&diff=867SLURM Compare2013-12-10T11:47:37Z<p>Bohme001: </p>
<hr />
<div>=== Rosetta Stone of Workload Managers ===<br />
<br />
PBS/Torque, Slurm, LSF, SGE and LoadLeveler [http://slurm.schedmd.com/rosetta.html Rosetta Stone]<br />
<br />
This table lists the most common command, environment variables, and job specification options used by the major workload management systems: PBS/Torque, Slurm, LSF, SGE and LoadLeveler. Each of these workload managers has unique features, but the most commonly used functionality is available in all of these environments as listed in the table. This should be considered a work in progress and contributions to improve the document are welcome.<br />
{| class="wikitable"<br />
!User Commands!!PBS/Torque!!Slurm!!LSF!!SGE!!LoadLeveler<br />
|-<br />
||Job submission|| qsub [script_file]|| sbatch [script_file]|| bsub [script_file]|| qsub [script_file]|| llsubmit [script_file] <br />
|-<br />
||Job deletion ||qdel [job_id]|| scancel [job_id]|| bkill [job_id]|| qdel [job_id]|| llcancel [job_id] <br />
|-<br />
||Job status (by job)|| qstat [job_id]|| squeue [job_id]|| bjobs [job_id]|| qstat -u \* [-j job_id]|| llq -u [username] <br />
|-<br />
||Job status (by user)|| qstat -u [user_name]|| squeue -u [user_name]|| bjobs -u [user_name]|| qstat [-u user_name]|| llq -u [user_name] <br />
|-<br />
||Job hold ||qhold [job_id]|| scontrol hold [job_id]|| bstop [job_id]|| qhold [job_id]|| llhold -r [job_id] <br />
|-<br />
||Job release|| qrls [job_id]|| scontrol release [job_id]|| bresume [job_id]|| qrls [job_id]|| llhold -r [job_id] <br />
|-<br />
||Queue list|| qstat -Q|| squeue ||bqueues ||qconf -sql|| llclass <br />
|-<br />
||Node list ||pbsnodes -l|| sinfo -N OR scontrol show nodes|| bhosts|| qhost|| llstatus -L machine <br />
|-<br />
||Cluster status|| qstat -a|| sinfo|| bqueues|| qhost -q|| llstatus -L cluster <br />
|-<br />
||GUI|| xpbsmon|| sview|| xlsf OR xlsbatch|| qmon|| xload <br />
|-<br />
|}</div>Bohme001https://wiki.anunna.wur.nl/index.php?title=SLURM_Compare&diff=866SLURM Compare2013-12-10T11:41:07Z<p>Bohme001: /* Rosetta Stone of Workload Managers */</p>
<hr />
<div>=== Rosetta Stone of Workload Managers ===<br />
<br />
PBS/Torque, Slurm, LSF, SGE and LoadLeveler<br />
<br />
This table lists the most common command, environment variables, and job specification options used by the major workload management systems: PBS/Torque, Slurm, LSF, SGE and LoadLeveler. Each of these workload managers has unique features, but the most commonly used functionality is available in all of these environments as listed in the table. This should be considered a work in progress and contributions to improve the document are welcome.<br />
{| class="wikitable"<br />
!User Commands!!PBS/Torque!!Slurm!!LSF!!SGE!!LoadLeveler<br />
|-<br />
||Job submission|| qsub [script_file]|| sbatch [script_file]|| bsub [script_file]|| qsub [script_file]|| llsubmit [script_file] <br />
|-<br />
||Job deletion ||qdel [job_id]|| scancel [job_id]|| bkill [job_id]|| qdel [job_id]|| llcancel [job_id] <br />
|-<br />
||Job status (by job)|| qstat [job_id]|| squeue [job_id]|| bjobs [job_id]|| qstat -u \* [-j job_id]|| llq -u [username] <br />
|-<br />
||Job status (by user)|| qstat -u [user_name]|| squeue -u [user_name]|| bjobs -u [user_name]|| qstat [-u user_name]|| llq -u [user_name] <br />
|-<br />
||Job hold ||qhold [job_id]|| scontrol hold [job_id]|| bstop [job_id]|| qhold [job_id]|| llhold -r [job_id] <br />
|-<br />
||Job release|| qrls [job_id]|| scontrol release [job_id]|| bresume [job_id]|| qrls [job_id]|| llhold -r [job_id] <br />
|-<br />
||Queue list|| qstat -Q|| squeue ||bqueues ||qconf -sql|| llclass <br />
|-<br />
||Node list ||pbsnodes -l|| sinfo -N OR scontrol show nodes|| bhosts|| qhost|| llstatus -L machine <br />
|-<br />
||Cluster status|| qstat -a|| sinfo|| bqueues|| qhost -q|| llstatus -L cluster <br />
|-<br />
||GUI|| xpbsmon|| sview|| xlsf OR xlsbatch|| qmon|| xload <br />
|-<br />
|}</div>Bohme001